Dynamic network component placement

ABSTRACT

A network component placement architecture addresses the difficult technical challenge of determining where specific technical components of an overall service may be provisioned. However, the placement decision can change over time in response to complex updates relating to service providers, the regions they support, the assets they support, and many other dynamic factors. Dynamic network component placement effectively determines updated possibilities for placing the technical components of a specified service, while meeting all of the applicable technical constraints on the placement of the technical components.

PRIORITY CLAIM

This application claims priority to provisional application Ser. No.62/088,474, filed 5 Dec. 2014, titled Hybrid Cloud Management, which isentirely incorporated by reference.

TECHNICAL FIELD

This application relates to analysis, control, and provisioning oftechnical components into a complex global network architecture ofvirtualized resources.

BACKGROUND

The processing power, memory capacity, network connectivity andbandwidth, available disk space, and other resources available toprocessing systems have increased exponentially in the last two decades.Computing resources have evolved to the point where a single physicalserver may host many instances of virtual machines and virtualizedfunctions. These advances had led to the extensive provisioning of awide spectrum of functionality for many types of entities into specificpockets of concentrated processing resources that may be locatedvirtually anywhere, that is, relocated into a cloud of processingresources handling many different clients, hosted by many differentservice providers, in many different geographic locations. Improvementsin cloud system control, deployment, and provisioning will drive thefurther development and implementation of functionality into the cloud.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of a global network architecture.

FIG. 2 illustrates an example implementation of a hybrid cloudarchitect.

FIG. 3 shows an example of type definition tables.

FIG. 4 shows logic for establishing type definition tables.

FIG. 5 shows database tables for equivalency mapping.

FIG. 6 shows logic for equivalency mapping.

FIG. 7 shows database tables for type-to-type translation.

FIG. 8 shows logic for type-to-type translation.

FIG. 9 shows a metadata architecture within the hybrid cloud architect.

FIG. 10 shows logic for metadata collection, creation, and derivation.

FIG. 11 shows another view of the metadata architecture within thehybrid cloud architect.

FIG. 12 shows placement pipeline circuitry.

FIGS. 13-19 show logic for determining feasible placement options fromcandidate placement options.

FIG. 20 shows another example of placement pipeline circuitry.

FIG. 21 shows another example of a placement pipeline circuitry.

FIG. 22 shows two placement pipelines working in sequence.

FIG. 23 shows an example of a hybrid cloud architect that supportsdynamic re-placement.

FIG. 24 shows an example of dynamic re-placement.

FIG. 25 shows a logical flow for dynamic re-placement.

FIG. 26 shows an example of offline dynamic re-placement.

FIG. 27 shows a cloud computing placement and provisioning architecture.

FIGS. 28 and 29 show logical flow for a cloud computing placement andprovisioning architecture.

FIG. 30 shows an example execution of the cloud computing placement andprovisioning architecture.

FIG. 31 shows an example baseline technical service template and aconcretized technical service template.

FIG. 32 provides an illustration of service provider metadata definingtypes, networks, and assets for a specific service provider.

FIG. 33 shows an example region roll-up.

FIG. 34 shows an example network roll-up.

FIG. 35 shows an example of network equivalence.

FIG. 36 shows an example asset roll-up.

FIG. 37 shows an example of asset equivalence.

FIG. 38 shows a VM resource definition in a baseline technical servicetemplate.

FIG. 39 shows a multiple stage type-to-type translation architecture.

FIG. 40 shows logical flow for multiple stage type-to-type translation.

FIG. 41 shows a cloud resource provisioning architecture with templateaggregation.

FIG. 42 shows a logical flow for a cloud resource provisioningarchitecture with template aggregation.

FIG. 43 shows an additional logical flow for a cloud resourceprovisioning architecture with template aggregation.

FIG. 44 shows an additional logical flow for a cloud resourceprovisioning architecture with template aggregation.

FIG. 45 shows another example of a cloud resource provisioningarchitecture with template aggregation.

DETAILED DESCRIPTION

FIGS. 1 and 2 provide an example context for the discussion of technicalsolutions for complex cloud architecture control and provisioningdescribed in detail below. The examples in FIGS. 1 and 2 show one ofmany possible different implementation contexts. In that respect, thetechnical solutions are not limited in their application to thearchitectures and systems shown in FIGS. 1 and 2, but are applicable tomany other cloud computing implementations, architectures, andconnectivity.

FIG. 1 shows a global network architecture 100. Distributed through theglobal network architecture 100 are cloud computing service providers,e.g., the service providers 102, 103, 104, 106, and 108. The serviceproviders may be located in any geographic region, e.g., United States(US) East, US West, or Central Europe. The geographic regions thatcharacterize the service providers may be defined according to anydesired distinctions to be made with respect to location. A serviceprovider may provide cloud computing infrastructure in multiplegeographic locations.

The service providers may provide computing resources via platforms thatare generally publicly available. Service providers may additionally oralternatively provide computing resources “on-premises”, which typicallyrefers to a location with increased privacy and security compared topublic cloud resources. An on-premises location may be within a securefacility owned by an entity which has moved computing functionality to acloud based implementation, for instance. Examples of service providersinclude Amazon, Google, Microsoft, and Accenture, who offer, e.g.,Amazon Web Services (AWS), Google Compute Engine (GCE), Microsoft Azure(Azure), and Windows Azure Pack (WAP) for on-premises cloudimplementations, as just a few examples.

Throughout the global network architecture 100 are networks, e.g., thenetwork 110, that provide connectivity within the service providers, andbetween the service providers and other entities. The networks 110 mayinclude private and public networks defined over any pre-determined andpossibly dynamic internet protocol (IP) address ranges. A hybrid cloudarchitect (HCA) 112 makes complex cloud architectural provisioning andexecution decisions across multiple cloud services, taking into accountthe global network architecture 100, the various service providerlocations and capabilities, and other factors. The provisioning andexecution decisions are discussed in detail below, and include, asexamples, determining what resources to instantiate, determiningplacement options for where (e.g., in which service provider regions) toinstantiate the resources, and determining possible alternativeimplementation options for the resources. Specific aspects of the HCA112 are described in more detail below.

As an overview, the HCA 112 may include metadata circuitry 114configured to collect, store, and analyze cloud service metadata. TheHCA 112 implements equivalency and type-to-type (TTT) circuitry 116 thatis configured to determine equivalency between assets and networkswithin resources, and map cloud resource types between disparate serviceproviders. A resource is a managed object, and types are prototypes ofthe managed objects. A ‘region’ may refer to a unit of hosting capacityin a particular geographic region, where types may be deployed.

The HCA 112 also includes placement circuitry 118 which is configured todetermine where, how, and with which service provider the functionalityrequested by a particular resource requester 150 may be instantiated inthe global network architecture 100. In other words, the HCA 112determines placement options for requested resources. The dynamicplacement circuitry 120 facilitates review and update of the placementcircuitry decisions. The HCA 112 may also implement an end-to-endprovisioning architecture 122 that is configured to, among otherfeatures, accept resource requester requests for cloud services,determine placement options, and execute provisioning actions once aplacement option is selected. The provisioning actions are described inmore detail below, and may include, as examples, determining whichresources to deploy, and providing instructions to resource providers toinstantiate the resources.

The actions taken by the HCA 112 are influenced by many technicalfactors, including metadata collected from various sources, includingservice provider metadata 152 that describes service provider offeringsand capabilities, and requester metadata 154 that describes the cloudfunctionality requests 156 made to the HCA 112 by the resource requester150, and the service requirements (e.g., PCI data compliance) for thefunctionality requests made by the resource requester 150.

In its role as the architect, the HCA 112 analyzes cloud servicerequests and makes decisions about implementation and provisioning ofthe requested services. This technical role is a complex one, due inpart to the disparate cloud computing services offered by each serviceprovider. That is, each service provider has a widely varying set oftechnical characteristics.

For instance, FIG. 1 shows a particular data center 124 for the serviceprovider 108 running many different virtual machines (VMs), each runningmany different virtual functions (VFs). The data center 124 may includea high density array of network devices, including routers and switches126, and host servers 128. The host servers 128 support a specific setof computing functionality that is offered by the service provider 108from the data center 124. As just one of many examples, the serviceprovider 108, through the data center 124 and its other infrastructure,may support many different types of virtual machines, differing bynumber of processors, amount of RAM, and size of disk, graphicsprocessors, encryption hardware, or other properties; multiple differenttypes of web front ends (e.g., different types and functionality forwebsites); several different types of database solutions (e.g., SQLdatabase platforms); secure data storage solutions, e.g., payment cardindustry (PCI) data (or any other secure data standard) compliantstorage; several different types of application servers; and manydifferent types of data tiers. Further, the service provider 108 and thedata center 124 may have further characteristics for the HCA to analyze,including whether the data center 124 is an on-premises or publiclocation; which networks can provide connectivity to the data center124; which assets the service provider 108 supports; and othercharacteristics.

FIG. 2 shows an example implementation of the HCA 112 configured toexecute complex cloud architectural provisioning and execution decisionsacross multiple cloud services. The HCA 112 includes communicationinterfaces 202, system circuitry 204, input/output interfaces 206, and adisplay 208 on which the HCA 112 generates a user interface 209.

The user interface 209 and the input/output interfaces 206 may include agraphical user interface (GUI), touch sensitive display, voice or facialrecognition inputs, buttons, switches, speakers and other user interfaceelements. Additional examples of the input/output interfaces 206 includemicrophones, video and still image cameras, headset and microphoneinput/output jacks, Universal Serial Bus (USB) connectors, memory cardslots, and other types of inputs. The input/output interfaces 206 mayfurther include magnetic or optical media interfaces (e.g., a CDROM orDVD drive), serial and parallel bus interfaces, and keyboard and mouseinterfaces.

The communication interfaces 202 may include wireless transmitters andreceivers (“transceivers”) 210 and any antennas 212 used by the Tx/Rxcircuitry of the transceivers 210. The transceivers 210 and antennas 212may support WiFi network communications, for instance, under any versionof IEEE 802.11, e.g., 802.11n or 802.11ac. The communication interfaces202 may also include wireline transceivers 214. The transceivers 214 mayprovide physical layer interfaces for any of a wide range ofcommunication protocols, such as any type of Ethernet, data over cableservice interface specification (DOCSIS), digital subscriber line (DSL),Synchronous Optical Network (SONET), or other protocol.

The system circuitry 204 may include any combination of hardware,software, firmware, or other logic. The system circuitry 204 may beimplemented, for example, with one or more systems on a chip (SoC),application specific integrated circuits (ASIC), microprocessors,discrete analog and digital circuits, and other circuitry. The systemcircuitry 204 is part of the implementation of any desired functionalityin the HCA 112. As just one example, the system circuitry 204 mayinclude one or more instruction processors 216 and memories 218. Thememory 218 stores, for example, control instructions 220 and anoperating system 222. The processor 216 executes the controlinstructions 220 and the operating system 222 to carry out any desiredfunctionality for the HCA 112. The control parameters 224 provide andspecify configuration and operating options for the control instructions220, operating system 222, and other functionality of the HCA 112.

The HCA 112 also includes storage devices (e.g., hard disk drives (HDDs)and solid state disk drives (SDDs)). For instance, the storage devicesmay define and store databases that the control instructions 220accesses, e.g., through a database control system, to perform thefunctionality implemented in the control instructions 220. In theexample shown in FIG. 2, the databases include a metadata database 226,an equivalency database 228, and a TTT database 230. Each of thedatabases 226, 228, and 230 define tables storing records that thecontrol instructions 220 read, write, delete, and modify to perform theprocessing noted below.

In that regard, the system circuitry 204, e.g., through the controlinstructions 220, may include metadata processing 232 configured tocollect, store, and analyze cloud service metadata; equivalency and TTTprocessing 234 that is configured to determine equivalency betweenassets and networks, including TTT processing configured to map cloudresource types between disparate service providers; a placement engine236 configured to determine where the functionality requested by aparticular resource requester may be instantiated in the global networkarchitecture 100; and dynamic placement instructions 238 configured toreview and update the decisions previously made by the placement engine236.

Equivalency and Type-to-Type (TTT)

The discussion below uses the example of a resource requester that hassubmitted a request for a bundle of services to be hosted in the cloud.The bundle of services may be defined by a service template thatidentifies the requested services, along with metadata that describesthe requested services. In this example, the bundle of services is for anew SharePoint site, which the service template defines as includingthree web front ends on three VMs, two application servers on two VMs,and a data tier of two SQL database servers on two additional VMs. Therequester metadata 154 indicates that the applications will work withPCI data, which calls for enhanced security and on-premises provisioningof the data tier, rather than provisioning into the public cloud.Further, the example assumes that the service template identifies publiccloud Blue VMs (from the hypothetical Blue service provider) as thebaseline template type for each of the VMs.

FIG. 3 shows an example of type definition tables 300, and FIG. 4 showsa corresponding logical flow 400 for establishing the type definitiontables 300. The type definition tables 300 may be manually populated,e.g., with expert determinations of how to map the parameters of oneresource to the parameters of another resource, and determinations ofwhich resources offered by a given provider are considered equivalent tothe resources offered by a different provider. In other implementations,automated analysis processes may add records to the tables 300, e.g., inresponse to real-world testing or pre-defined rules that determine whentwo resources offered by different service providers are equivalent, andhow the parameters map between the resources. The same types ofautomated testing/rule based, and expert manual processes may also addrecords to the equivalency mapping tables shown in FIG. 5, and thetranslation tables shown in FIG. 7, which are discussed below.

The TTT processing 234 defines a type table 302 (402) and populates thetype table 302. The type table 302 includes, e.g., a type name field 304(404) and a type identifier field 306 (406). In this example, the typetable 302 defines four VMs types from four different service providers:Blue, Green, Black, and Red. Each VM type has been assigned a typeidentifier (408), for instance the Blue VM is type 2, and the Red VM istype 5. The type table 302 may define and identify any number of VMs ofdifferent types. In addition, the type table 302 may define and identifyany number and type of other technical components of a computing serviceto be provisioned in the cloud. For example, the type table may defineand assign types to websites, storage accounts, networks, load balancinghardware, databases, monitoring systems, or any other type of technicalcomponent that serves the same function in different service providersystems.

FIG. 3 also shows a type properties table 320 that the TTT processing234 defines (420) and populates. The type properties table 320 includes,e.g., a property field 322 (422) and a type identifier field 324 (424)for establishing the various properties that characterize any given typefor any given service provider. That is, the type properties table 320links types to properties (426). In this example, the type propertiestable 320 links the Red VM, type 5, to properties: VM Name, Processors,RAM, Disk Size, and OS Disk. The type properties table 320 links theBlue VM, type 2, to properties: Identifier, Size, and OS Disk.

The type properties table 320 may also include a property type field326. The property type field 326 may include an identifier for eachproperty that provides additional information of the type of thatproperty (428). For instance, the property type for OS Disk is set to 2,which indicates in this example that OS Disk is an ‘asset’, and may besubject to equivalence mapping as described below. Similarly, theproperty type for Network Name is set to 1, which indicates that NetworkName is a ‘network’, and may also be subject to equivalency mapping. Theother property types may be set to NULL to indicate that no specialprocessing (e.g., equivalency mapping) is applied to them prior to TTTtranslation.

In one implementation, the TTT processing 234 is implemented with anasset equivalency mapping followed by a TTT translation. Regarding assetequivalency mapping, for instance, the TTT processing 234 may determinean asset, e.g., OS Disk, of the first component type, and an assetvalue, e.g. “DiskA-27.vhd” for the asset. The TTT processing 234 maythen determine an asset substitution, e.g., GUID3 for the asset, forprovisioning the asset in the second service provider. The TTTprocessing 234 then replaces the asset value with the assetsubstitution, e.g., in the bundle of data defining the services toprovision, such as in a Java Script Object Notation (JSON) file. In thatregard, the equivalency mapping is configured to determine which serviceproviders offer equivalent assets to the baseline assets specified,e.g., in a technical service template, and may provide identifiers ofthe service providers to other processing circuitry in the HCA 112, suchas the placement engine 236. Once the equivalency mappings are executed,the TTT processing 234 performs TTT translation.

FIG. 5 shows database tables for equivalency mapping 500, with FIG. 6providing a corresponding logical flow. Continuing the example aboveregarding the SharePoint site, the TTT processing 234, reads themetadata to determine to place the data tier in a PCI compliant cloudservice. The metadata architecture, including the sources of metadataand how it is stored, is described in detail below. As such, the TTTprocessing 234, staring with the template Blue VM and OS Disk (e.g., forSQL server for the data tier), searches for an equivalent OS Disk for asecure environment.

The equivalence mapping may execute for any asset included in theresource requester request for a bundle of services, such as disk imagesand also for networks. The equivalence mapping may be a single asset tosingle asset translation stage, pre-defined for specific assets.

FIG. 5 shows an asset equivalence table 502 (602) and an asset table 504(604) for use in equivalence mapping for assets. The asset equivalencetable 502 stores a unique identifier for the asset, and groups assetstogether as being functionally equivalent. The asset table 504 stores anasset equivalency identifier 506 and an asset name 508. The value forthe asset equivalency identifier 506 in the asset table 504 is a foreignkey to the value in the asset equivalence table 502. Thus, when specificasset share the same asset equivalency identifier 506, those specificassets are defined as equivalent assets by the tables 502 and 504. Theasset name 508 may provide asset values that can be used for assetsubstitutions. For OS Disk assets, the asset name 508 may provide alocation, e.g., a file path or globally unique identifier (GUID), atwhich to find a disk image.

The equivalence mapping process obtains the asset name specified, e.g.,“Disk A-27.vhd”, for the OS Disk asset in the template VM (606). Theequivalence mapping performs a lookup on the asset table 504 with theasset name (608), and obtains the records from the asset table with thematching asset equivalency identifier 506 (610). In this case, theresults are “abc.ami” and GUID3. The equivalence mapping determines aregion for each result, e.g., by searching a region table (612), anddetermines which regions are compatible with provisioning the resource,e.g., based on the metadata (614). The result in this example is GUID3,which corresponds to a disk image in the Red VM on-premises region(616). In other words, the equivalence mapping process has determinedthe asset substitution GUID3 for the asset name “Disk A-27.vhd”. Havingdetermined GUID3 as the asset substitution, the equivalency mappingreplaces “Disk A-27.vhd” with GUID3 (618).

For networks, the equivalence mapping takes the value of the NetworkName (620), e.g., “Network 1” for the Blue VM template. The equivalencemapping performs a lookup of the network name value in a network table520 to find the parent network (622), e.g., Parent Network A. Thenetwork table 520 defines Networks 1, 2, and 3 as roll-up members of theparent network A (624). The members were added due to their equivalence,and thus the equivalence mapping may select from Network 2 (Greenregion) or Network 3 (Red region) as a substitution for Network 1. Inthis example, the equivalence mapping selects Network 3 as belonging toa region compatible with PCI data (626), and makes the assetsubstitution by replacing Network 1 with Network 3, e.g., in the JSONdescription of the service request (628).

Each network might have, for instance, a different IP address range, butfor the purposes of determining equivalence, any of Networks 1, 2, and 3are equivalent to each other, because they all belong to Network A. Inthat respect, Network A is an abstraction in the architecture that thearchitecture may use to attach custom metadata to actual virtualnetworks that are the children of Network A, and that are defined to beequivalent by virtue of their inclusion under Network A. Roll-upnetworks may be nested inside one another as well. Each network within aset of networks within a specific network may be considered equivalent.Network equivalency may determine network options that place the networkin a different region than that specified in the technical resourcetemplate. Expressed another way, network equivalency defines equivalencebetween multiple networks from multiple providers. The equivalencyanalysis makes the equivalency decisions automatically, rather thanbombard a user with questions. When multiple network options areavailable, the equivalency processing may make a selection based on aprecedence order defined and linked to the networks or assets, forinstance.

After the assets, networks, and other special types are mapped, the TTTprocessing 234 proceeds with TTT translation (630). As one aspect, theTTT processing 234 translates baseline technical component types tosubstitute technical component types, e.g., when the baseline technicalcomponent type may be implemented by a different service provider thatdefines a different type that performs equivalent functionality. In thatregard, the TTT processing 234 is configured to determine which serviceproviders offer equivalent types to the baseline type, as describedabove with respect to FIGS. 1-6, and may provide identifiers of theservice providers to other processing circuitry in the HCA 112, such asthe placement engine 236.

FIG. 7 shows database tables for TTT translation 700, with FIG. 8providing a corresponding logical flow. In one implementation, the TTTtranslation references an input table 702, a translation table 704, andan output table 706. The input table includes a translation identifier708, an input property identifier 710, and an input parameter name 712.The input property identifier 710 corresponds to the propertiesidentified in the type properties table 320, the translation identifier708 provides a translation path to follow as will be explained morebelow. The input parameter name 712 specifies an input parameter to ascript (if any) to execute to assist with TTT translation.

The translation table 704 includes a translation identifier 714 to matchagainst the translation identifier 708, and a path field 716. The pathfield 716 specifies a script to execute, if any, to facilitate TTTtranslation, taking input from the input parameter identified in theinput table 702. The path field 716 may specify the script by providinga path to the script and a name for the script in a given file system.The scripts may be implemented in a wide variety of scripting languages,including PowerShell, Ruby, or Python, e.g., by resource translationexperts who determine how to map parameters back and forth betweenspecific resource types. The output table 706 includes a translationidentifier 718, and an output identifier 720. The output identifier 720specifies an output property to which the input property maps. Thetranslation table 704 links the input table 702 and the output table 706through the translation identifier 714.

The particular example given in FIG. 7 is specific to several examplesof translating between Blue VMs, Type 2, to Red VMs, Type 5. Similartables may be prepared for translating properties between any othertypes defined in the type table 302. Furthermore, any of the tables usedfor type-to-type translation and equivalency determinations may beresource requester specific. In other words, any particular resourcerequester may control or specify to the HCA 112 how to translate typesfor that particular resource requester, and which assets are consideredequivalent for that particular resource requester. Accordingly, the TTTcircuitry 116, in its analysis, may access and retrieve data from tablesin the equivalency database 228 and TTT database 230 responsive to thespecific entity that is requesting the hosted services.

In some implementations, the TTT circuitry 116 performs translation to afinal type through a reference type. The two step translation avoids theexponential increases in translation tables and the associatedcomplexity and memory requirements that would be defined for allpossible combinations of direct translation from ‘n’ types to any of‘n−1’ other types. FIGS. 39 and 40 provide additional details of the twostep translation.

FIG. 39 shows a multiple stage type-to-type translation architecture3900. First, however, FIG. 39 shows one possible baseline approach 3902.FIG. 39 compares the baseline approach 3902 to the two step translationmodel 3918 preferably executed by the TTT circuitry 116 in thetranslation architecture 3900. In this example, there are five differentVM types defined from different service providers and that havedifferent characteristics: a Red VM 3906, a Blue VM 3908, a Green VM3910, a White VM 3912, and a Black VM 3914. The VM types differaccording to how they parameterize their hardware feature set. In someinstances, such as for the White VM 3912, the type includes specificparameters for number of processors and amount of RAM. In otherexamples, such as for the Black VM 3914, the feature set is representedin a text string, e.g., “High Performance”. The baseline approachdefines a set of translation and equivalency tables in the equivalencydatabase 228 and TTT database 230 for directly converting from any ofthe five types to any other of the five types. That is, each of the fiveVMs has four sets of translation tables 3916, leading to a significantinvestment in underlying preparation time, resource consumption, andinfrastructure for translation.

FIG. 39 also shows how the multiple stage architecture 3900 defines atwo-step translation reference model 3918, and FIG. 40 shows acorresponding logical flow 4000 for multiple stage type-to-typetranslation. The types to include in the reference model 3918 areidentified (4002). The reference model 3918 designates a specific typeas the reference type (4004). In the example shown in FIG. 39, the GreenVM 3910 is chosen as the reference type 3920. The reference type 3920may be any selected type. In some implementations, the reference type3920 is chosen to be the type most commonly represented type in thetechnical service templates 908.

The HCA 112 sets up translation and equivalency tables for each type tothe reference type 3920 (4006). Similarly, the HCA 112 sets uptranslation and equivalency tables from the reference type 3920 to eachother type (4008). As indicated by the two-step translation referencemodel 3918, a conversion from a Red VM type to a Black VM type passesthrough the reference type 3920. The translation is from source type,the Red VM 3906, to the reference type 3920 (the Green VM 3910), andthen from the reference type 3920 to the destination type, the Black VM3914.

The TTT circuitry 116 determines a source type to translate (4010) and adestination type to which to translate (4012). If they are the same,then no translation is needed (4014). Otherwise, when the source type isthe reference type 3920, then the TTT circuitry 116 preforms a singlestep translation from the reference type to the destination type (4016).When the destination type is the reference type, the TTT circuitry 116also performs a single step translation from the source type to thereference type (4018). When the reference type is neither the sourcetype nor the destination type, then the TTT circuitry 116 performs atwo-step translation: first from the source type to the reference type(4020), then from the reference type to the destination type (4022).

That is, the two-step translation model 3918 sets up a mechanism bywhich, at most, the TTT circuitry 116 performs two translations to movefrom a source type, e.g., specified in a baseline technical servicetemplate, to a destination type to be deployed in a selected location.The two-step translation reference model 3918 achieves a significantdecrease in the underlying preparation time, resource consumption, andinfrastructure for translation between types. The reference model 3918avoids the exponential increases in translation tables and associatedcomplexity and memory requirements that would be defined for allpossible combinations of direct translation from ‘n’ types to any of‘n−1’ other types.

Several examples follow with reference to FIGS. 7 and 8 concerningconverting from Type 5 to Type 2. The TTT translation finds in the typeproperties table 320 the next property to translate for the type beinganalyzed. The next property in this example is the VM Name property,property identifier 1 (802). The TTT translation searches the inputtable 702 with the property identifier 1 as the input property ID (804)and thereby locates the translation(s) identified by translationidentifier 1 (806). In this instance, the translation is the singleinstance of translation identifier 1.

Next, the TTT translation searches the translation table 704 with thetranslation identifier of 1 to determine whether to execute a script(808). In this instance, the path field 716 is NULL, signifying thatthere is no script to run. The TTT translation also searches the outputtable 706 with the translation identifier of 1 to find the correspondingoutput identifier (810). In this case the output identifier is 6,corresponding to the Identifier field as noted in the type propertiestable 320. Because there is no script to execute, the TTT translationdirectly copies the value form input property 1, VM Name, into outputproperty 6, Identifier. That is, in the Type 2 VM, the Identifier fieldstores the value that the Type 5 VM stores in its VM Name field.

Similarly, in converting from Type 2 to Type 5, the input property willat some point be Identifier, property 6. The input table 702 identifiestranslation identifier 3 for this input property. Translation identifier3 has no script identified in the translation table 704, and has anoutput property of 1, VM Name, as identified in the output table 706.According, the TTT translation copies the value of the Identifierproperty directly into the VM Name property when converting from Type 5to Type 2.

The process repeats for each property (814). After each property istranslated, the TTT translation has produced a translated object thatmay be provided to subsequent processing, e.g., a provisioning engine(816).

Taking another example, the next property is Processors, propertyidentifier 2. The TTT translation finds two instances of a matchingtranslation identifier of 2 in the input table 702. The two instances oftranslation identifier 2 reference the Processors property, ID 2, andthe RAM property, ID 3. In addition, the translation table 704 indicatesto run ‘script1’ for translation identifier 2, and the output tableindicates to place the output into output identifier 7, the Sizeproperty for the Type 2 VM. Accordingly, the TTT translation extractsthe Processors and RAM property values from the template and providesthe Processors and RAM property values as parameters to the script(818), determines the destination property (820), and executes thescript which writes the script output to the destination property (822).In this example, the script accepts the Processors and RAM values fromthe input properties, and outputs a value for the Size propertycorresponding to the Processors and RAM values. For instance, ifProcessors is the value 4 and RAM is “8 GB”, then the script maydetermine that the Size is ‘Standard A1’ and output, e.g., {“Size”:“Standard A1”} as a JSON conversion for obtaining a Type 2 equivalent VMproperty for the Type 5 number of processors and amount of RAM. Thescript may implement any such pre-defined mapping of input variables tooutput variables.

Similarly, in converting from Type 2 to Type 5, the input property willat some point be property 7, Size. The input table 702 specifies atranslation identifier of 4 for the Size property, and that an inputparameter called SizeInput is used by a script to run for thetranslation. The translation table 704 indicates that the name of thescript to run is ‘script2’, and the TTT translation executes the scriptwith the SizeInput set to the value of the Size property, e.g.,‘Standard A1’. The script implements a predetermined mapping of the Sizeproperty to the output parameters 2 (Processors) and 3 (RAM) asidentified in the output table 706. In this instance, the scripttranslates ‘Standard A1’ to the value ‘4’ for the processors propertyand the value ‘8 GB’ for the RAM property. That is, the TTT translationconverts the single property {“Size”: “Standard A1”} to two properties:{“Processors”: 4}, and {“RAM”: “8 GB”}.

Expressed another way, the TTT circuitry 116 includes type definitioncircuitry configured to assign (e.g., via the type table 302) a firsttype specifier (e.g., Type 5) to a first component type (e.g., Blue VMs)available from a first service provider, and assign a second typespecifier (e.g., Type 2) to a second component type (e.g., Red VMs)available from a second service provider.

The TTT circuitry 116 also includes property linking circuitryconfigured to link (e.g., via the type properties table 320) a first setof technical properties (e.g., Processors and RAM) to the firstcomponent type and link a second set of second technical properties(e.g., Size) to the second component type. Property translationcircuitry establishes a translation correspondence (e.g., via the inputtable 702, translation table 704, and the output table 706) between thefirst set of technical properties for the first component type and thesecond set of technical properties for the second component type.

Type mapping circuitry is configured to make equivalency substitutions,by determining a first asset (e.g. OS Disk) of the first component type,and an asset value for the first asset (e.g., “Disk A-27.vhd”). Themapping circuitry also determines an asset substitution (e.g., GUID3)for the first asset, for provisioning the first asset to the secondservice provider. The mapping circuitry also replaces the asset valuewith the asset substitution. After the equivalency substitutions, thetype mapping circuitry translates the first component type into thesecond component type according to the translation correspondence. As aresult, the type mapping circuitry prepares a technical description(e.g., a JSON document) for provisioning the first component type at thesecond service provider as the second component type.

Execution of the TTT circuitry 116 may follow, e.g., a placement enginethat determines in which regions cloud resources that implement afunctionality request may be instantiated. When the resource requester150 makes a decision on region, the TTT circuitry 116 may then translatethe resource template descriptions for the cloud resources forcompatibility with the service provider hosting the services in thatregion. If the cloud resources will be deployed to the region andservice provider already specified in the resource template, then notranslation needs to be performed.

Returning to the SharePoint example, the service template defined threeweb front ends on three Blue VMs, two application servers on two BlueVMs, and a data tier of two SQL database servers on two additional BlueVMs. The requester metadata 154 indicated that the applications willwork with PCI data, which calls for enhanced security and on-premisesprovisioning of the data tier, rather than provisioning into the publiccloud. As such, the TTT translation converted the data tier from BlueVMs to Red VMs which, through the metadata, are known to be PCIcompliant.

At deployment time, the service template will specify three web frontends in Blue VMs, and two application servers in Blue VMs, all connectedto the same network, Network 1. However, the two VMs for the data tierare in Red VMs with a different servicer provider under Network 3. ButNetwork 1, Network 2, and Network 3 were defined under the same ParentNetwork A, indicating that all three networks can communicate with oneanother, allowing the complete set of VMs to interoperate as needed.

Metadata

The HCA 112 implements a metadata architecture that helps address thetechnical challenge of finding viable placement options for implementingtechnical service requests. The metadata architecture links varioustypes of metadata to technical components, e.g., types and assets, totechnical service templates, and to a container hierarchy. The HCA 112injects specific metadata subsets into a placement analysis pipelinethat determines where the technical components that make up the servicerequest may be placed in the extensive and complex service providerspace.

FIG. 9 shows a metadata architecture 900, including an exampleimplementation of metadata circuitry 114 within the hybrid cloudarchitect 112. The metadata database 226, in this implementation, storesrequester metadata 902 characterizing the resource requester servicerequest, e.g., what, if any, data security features does the requestedservice need; service provider metadata 904 characterizing the serviceprovider capabilities with respect to technical component types, assets,region characteristics, and other service provider aspects; and metadatafor the container metadata 906, that characterizes the sections,technical component types 910 (e.g., VMs, websites, and DBs), assets 912(e.g., OS disks) networks, and other features of the technical servicetemplates 908 that define a baseline implementation for the availableservice sets that a resource requester may order. The technical servicetemplates 908 may also be referred to as catalog items. A templatesdatabase 909 may store the technical service templates 908. In thatregard, the templates database 909 may provide a pre-defined library oftechnical service templates 908, each of which provides an initial orbaseline specification of one or more resources (e.g., a VMs, DBs, andnetworks) that implement a technical service request (e.g., a SharePointsite), including the parameters for the resources (e.g., the size of theVM), and placement options (e.g., to be placed as a default in the Blueprovider E.U. north region). The HCA 112 may use additional, different,or fewer types of metadata in other implementations.

FIG. 10 shows a corresponding logical flow 1000 for metadata collection,creation, and derivation. The communication interface 202 receives,e.g., from a service provider or another metadata source, serviceprovider metadata 904 that characterizes a virtualized hosting regioncontrolled by the service provider (1002). The service provider metadata904 may describe, as just a few examples, the technical component typessupported by the service provider in the service provider regions, theassets supported by the service provider; the types of data securityavailable from the service provider; which resource requesters havesubscriptions to the service provider; for which service providerregions, networks, or other features the subscriptions apply, andwhether the service provider regions are public or private (e.g.,on-prem regions).

The communication interface 202 also receives, e.g., from a resourcerequester 150, requester metadata 902 (1004). The requester metadata 902may be provided by a particular employee at the resource requester 150who is submitting the resource request, may be automatically provided bythe resource requester processing systems (e.g., by providingpre-established metadata for particular resources commonly requested bythe resource requester), or in other ways. The requester metadata 902characterizes a technical service request made by the resource requester150 for virtualized hosting, e.g., a request for a new toy developmentenvironment. As a few examples, the requester metadata 902 may indicatewhich, if any, aspects of the resource requester service request havespecific data security requirements, e.g., requirements for PCIcompliance; how many users are expected to use the servers, programs,and databases in the development environment; where the users reside andfrom where they are expected to access the services (this may driveplacement decisions for locating technical component types in regionsclose to the employees, for instance, or as another example, ensuringthat technical components that handle data on a European Union (EU)citizen are placed within EU boundaries and meet all EU data handlingrequirements); the level of criticality of the development environment;applicable service level objectives (SLOs) and service level agreements(SLAs); and other resource requester specific aspects of the technicalservice request. The requester metadata 902 may also characterize theresource requester itself, including, as one example, identifiers of theservice providers, service provider regions, and service providernetworks to which the resource requester 150 has active subscriptions.Given the potentially immense array of possible placement options, themetadata architecture 900, in conjunction with the processing describedabove and below, significantly increases the efficiency with whichplacement options are identified.

To obtain the requester metadata 902, the HCA 112 may present theresource requester 150 with a series of metadata questions for theresource requester 150 to answer, e.g., through a metadata completiontemplate 916 generated in the GUI 209 and displayed locally at theresource requester 150. The metadata architecture 900 may store theenterprise metadata 902 in many different manners in the metadatadatabase 226. As one example, the enterprise metadata 902 may take theform of tag and value pairs, e.g., {“Number of Users”, “500”}, or {“DataType”, “PCI”}, in XML, JSON, or another format, or as data recordsstored in a database with columns pre-defined to hold the metadataanswers for each metadata question. That is, the technical servicetemplates 908 may broadly apply across a wide range of implementations,with customization performed in response to the specific requestermetadata 902. In that respect, the HCA 112 may include mapping rules914. The mapping rules 914 obtain derived metadata from, e.g., therequester metadata (1006). The mapping rules 914 may also specifystoring the derived metadata into specific parameter fields of thetechnical service template for the service request made by the resourcerequester 150. As one example, a mapping rule may convert a resourcerequester metadata answer of “300 expected users” into derived technicalmetadata of a VM Size of “Standard A0” or “4 Processors, 8 GB RAM”, andsave the derived metadata into the technical service template that theplacement circuitry 118 will process for the particular technicalservice request. A technical service template with its variableparameter fields completed may be referred to below as a ‘concretized’template (1008).

The mapping rules 914 generate additional technical metadata, e.g., fromthe resource requester metadata 902. The additional technical metadatabecomes part of the concretized technical service template forconsideration by other processes in the HCA 112, including the placementengine 236. For instance, a mapping rule may specify that the enterprisemetadata 902 of {“Number of Users”}>200 maps to additional technicalmetadata such as “{Size, A1}” or {“Processors”: 4}, and {“RAM: 8 GB”}.This rule avoids asking the resource requester a highly technicalquestion that they are unlikely to understand or have an answerfor—namely how to specify a particular size of VM for a given serviceprovider. The rule translates the more understandable answer concerningnumber of users into the technical size specification of a VM asunderstood by the service provider. As such, the placement engine 236has additional information on which to make placement decisions, whilemaintaining the specific requester metadata 902 separately from theadditional technical metadata that may be inserted into the template.

The metadata database 226 may also define a container metadata 906(1010). FIG. 11 shows another view of the metadata architecture 1100within the hybrid cloud architect 112, with additional detail of thecontainer metadata 906. The container metadata 906 defines a view of theresource requester and the types of structures applicable to theresource requester and its activities. That is, the container metadata906 is a pre-defined real-world mapping of the operational structure ofa particular resource requester to a metadata hierarchy. An example isgiven below of a toy company that uses particular services andenvironments (e.g., test and development environments). However, thecontainer metadata 906 may of course change to align with any particularinternal structure of a given resource requester, for instance, byadding or removing new environments specifically used by that resourcerequester. As such, the implementation of the container metadata 906 mayvary widely between resource requesters. The container metadata 906serves the technical purpose of defining relationships between resourcesand the containers or owners of those resources, prior to provisioning.The metadata architecture thereby facilitates a metadata driven policyplacement that solves the difficult technical challenge of findingplacement options for complex technical service requests in an automatedway, and without subjecting the resource requester to repetitive trialand error approaches to tweaking service requests to find a successfulplacement.

FIG. 11 shows an example of a toy company container hierarchy 1102. Thecontainer hierarchy 1102 defines a multiple level technical containerstructure (1010). The structure includes container levels thatoptionally inherit properties from prior container levels. Eachcontainer level may be populated with specific container metadata(1012), e.g., as a pre-execution step performed by individuals havingknowledge of the internal structure of the resource requester; as anautomated step using pre-defined metadata values assigned to themultiple level technical container structure, and established, e.g.,when the multiple level technical container structure was designed andimplemented for the resource requester; or as a combination of automatedand manual metadata entry. An example of a multiple level technicalcontainer structure is provided below.

The container hierarchy 1102 includes a resource requester level 1104.At the resource requester level 1104, the container metadata maydescribe aspects of the resource requester in general, or as a whole.For instance, the container metadata may describe the type of resourcerequester, its products, locations, number of employees, employeelocations, and other resource requester characteristics.

In the example of FIG. 11, the container hierarchy 1102 includes aservice level 1106 as the next container level. The service level 1106may represent particular functions or processes in place within theresource requester, e.g., new toy design, research and development, toymarketing, and toy inventory and freight logistics. In otherimplementations, the container hierarchy 1102 may define a resourcerequester unit or division level as the next container level instead. Inthe service level 1106, the attached container metadata may include, asjust a few examples, a level of criticality for the service (from whichthe mapping rules 914 or placement engine 236 may derive or imply SLAsand SLOs and affect placement decisions, for instance); cost centersthat pay for the service; administrator IDs; service descriptions; costapproval personnel; dollar thresholds for automatic approval withoutcontacting the cost approval personnel; available budget for theservice; employment restrictions; and more specific SLOs and SLAs (e.g.,that roll-up into the SLOs and SLOs from the level above).

The container hierarchy 1102 also includes an environment level 1108.The environment level 1108 may define specific operational types thathelp provide the services defined at the service level 1106 for the toycompany. As examples, the operational types may include productionenvironments, test environments, and development environments. Thecontainer metadata attached at the environment level 1108 may include,as examples, a description or identification of the environment (fromwhich the mapping rules 914 or placement engine 236 may derive or implyadditional metadata affecting placement decisions, e.g., data securityrestrictions on production environments); identification of regulatoryissues and data security requirements (e.g., compliance with PCI, PII,or ITAR); the owner of environment; charge codes, budget allocation, orother financial characteristics; and more specific SLOs and SLAs (e.g.,a more specific level of availability or reliability for the productionenvironment).

The topology level 1110 may include topology metadata that identifies arelated group of resources at the resource level 1112. For instance, atopology group of resources may be defined to include members thatcorrespond to a collection of resources implemented by a particularservice provider. That is, the topology level 1110 may establish acollection of resources having a predefined meaning to the serviceprovider. As one example, the topology metadata may define a Sharepointsite as a collection of several VMs, DBs, and a connecting network.

The resource level 1112 represents specific technical components thatimplement a topology and an environment. For instance, the resourcelevel 1112 may include container metadata that specifies properties fortechnical component types, such as VM properties, e.g., properties forsize, processors, RAM, or other hardware components, databaseproperties, or web front end properties; properties for networks;properties for assets, such as names or other identifiers for websitesand disk images.

Any of the metadata components of the container hierarchy and any fieldsof the technical service templates may be pre-defined and fixed or maybe variable parameter fields. The HCA 112, e.g., via the mapping rules914, may derive a technical component value from any portion of therequester metadata 902, existing container metadata 906, or serviceprovider metadata 904, and store the technical component value in any ofthe parameter fields, whether in the technical service templates 908 orin the container hierarchy. Accordingly, when the resource requester 150requests implementation of a technical service, the HCA 112 may retrievethe baseline technical service template pre-defined for that particulartechnical service, populate parameter fields specific to the resourcerequester 150 according to the metadata, and pass the specific template(and the metadata) to the placement engine 236 for determining placementoptions. That is, while pre-defined technical service templates areavailable and specify one possible baseline implementation for one ormore resources, that baseline template service template changes to aspecific template according to the particular resource requester and themetadata. For example, the baseline technical service template mayinclude an empty parameter field for number of users, or size. The HCA112 creates the specific template by inserting, e.g., matching instancesof provider metadata 904, into the baseline template to obtain thespecific template, also referred to as a concretized technical servicetemplate.

As a specific example, the technical service template for a developmentenvironment for the toy company may define a webserver, applicationserver, and a database as the technical component types that makeup thedevelopment environment. The technical service template may furtherspecify assets. Examples of the assets include a deployment package thatdeploys content onto webservers and into SQL databases, and OS diskimages specified by image names for the images that provide thewebserver, application server, and database functionality.

Expressed another way, in some implementations, the technical servicetemplates 908 are hierarchical files, e.g., JSON files. The filesspecify and identify each resource, the relationship between resources,and the technical metadata for the resources. The technical servicetemplates 908 may include parameterized values. The requester metadata902 and service provider metadata 904 provide sources of metadata forderiving additional metadata. The derived metadata may be stored in thefields for the parameterized values.

In addition, the HCA 112 may derive implementation aspects from therelationships between resources. For instance, a technical servicetemplate may indicate that a database is used by a website, and thatboth are part of an application. The HCA 112 may automatically derive amonitoring relationship and monitoring implementation for the databaseand website in response. That is, knowing the relationships allows theHCA 112 to determine, e.g., which resources to monitor together, given,e.g., likely operational and failure interrelationships. As one example,a technical service template may specify that a web server relies on aparticular database and a particular network. The defined relationshipof the database to the web server and the network to the web serverallows the HCA 112 to prioritize a troubleshooting analysis for the webserver to the database and network resources.

That is, the HCA 112, using metadata obtained prior to provisioning,initiates execution of a placement engine 236 (e.g., implemented as aplacement analysis pipeline) on the concretized technical servicetemplate. The service provider metadata, container metadata, andrequester metadata are inputs to the placement engine 236 and availableat all pipeline stages, to determine feasible placement options forimplementing the technical service request (1014). One technicaladvantage is that the placement engine 236 has the technical dataavailable to it for deciding placement options in a very complex fieldof service providers, and for automatically determining options forplacement that are not literally specified in the baseline technicalservice template. The placement pipeline circuitry may implement asequence of pipeline stages. Each pipeline stage is a localized unit ofprocessing that accepts data inputs and produces data outputs based onthe specific set of processing tasks allocated to and implemented inthat particular pipeline stage.

Placement Engine and Re-Placement

FIG. 12 shows an example implementation of a placement engine 1200 bythe placement circuitry 118. FIGS. 13-19 show corresponding logicalflows for the placement engine 1200. The placement engine 1200 takes asinput a technical service template 1202 which is typically fullyconcretized with respect to the particular resource requester 150 andall or part of the metadata in the metadata database 226. The placementengine 1200 starts with a set of candidate placement options 1204 foreach resource defined in the technical service template 1202 anddetermines the feasible placement options 1206 for each resource. In oneimplementation, the placement engine 1200 eliminates, stage-by-stage,placement options from the candidate placement options 1204 that do notpass the filter defined in any particular stage.

The placement engine 1200 also performs filtering to impose an ordering(e.g., by cost, usage popularity, reliability, or other metric) thatresults in an ordered set of feasible placement options 1208. Theplacement engine 1200 generates a GUI 1210 which the resource requester150 renders on a display 1212. The GUI 1210 presents the ordered set offeasible placement options for selection by the resource requester 150.The selection of placement options may drive TTT processing to convertthe baseline technical service template into a specific service templatefor the resource requester 150 and for the technical services theresource requester 150 requested.

The placement engine 1200 performs a placement analysis for eachresource 1214 defined in the technical service template 1202. One aspectof the placement engine 1200 is hard technical decision processingstages that make specific determinations on whether specific serviceprovider regions are feasible placement options. In that regard, the HCA112 may define roll-up regions (some examples are described below inFIGS. 32-37) including two or more regions that the HCA 112 considersequivalent, e.g., with respect to geographic location or connectivityspeed and reliability. That is, regions may be members of regioncollections, and each region in the collection may be consideredequivalent such that if one region is a feasible placement option, theother regions in the collection are also feasible placement options.

Another aspect of the placement engine 1200 is a metadata processingstage. The metadata processing stage may make resource requesterspecific placement determinations. These determinations may turn on therequester metadata 902. For instance, regions that cannot meet the datasecurity requirements specified by the resource requester 150 may beeliminated from consideration. That is, the metadata processing stagemay include resource requester specific rulesets that encode resourcerequester policies, e.g., data governance policies and employee locationpolicies that affect placement decisions.

In the example shown in FIG. 11, the processing aspects of the placementengine 1200 are implemented by the placement pipeline circuitry 1216.The placement circuitry 118 includes multiple sequential pipelineprocessing stages. There may be additional, different, or fewer pipelineprocessing stages, and the pipeline processing stages may be arranged ina different sequence than shown in FIG. 12 and with different logicalflows shown in FIGS. 13-19.

The placement pipeline circuitry 1216 includes a subscription stage 1220configured to determine to which service provider regions and networksthe resource requester 150 has active subscriptions. The subscriptionstage 1220 receives the initial set of candidate placement options 1204,e.g., the set of service provider regions known to the HCA 112 (1302),and receives the next resource to analyze in the technical servicetemplate 1202 (1304). The subscription stage 1220 also receives metadatathat characterizes to which regions the resource requester currently hasactive subscriptions (1306). This may include the requester metadata 902and the service provider metadata 904, as examples. The subscriptionstage 1220 determines which of the candidate placement options areactually available to the resource requester in view of the subscriptioninformation (1308), and eliminates from further consideration thoseregions that are not subscribed (1310). The elimination may happenbecause, e.g., the resource requester 150 does not subscribe to theservice provider at all, because the resource requester 150 does notsubscribe to any service provider networks currently offered, or forother subscription reasons. The subscription stage 1220 communicates theupdated set of placement options to the next processing pipeline stage,the type stage 1222 (1312).

Expressed another way, associated with the resource requester 150 aresubscriptions or accounts, e.g., to cloud service providers such asAmazon or Microsoft. If the resource requester 150 does not have asubscription with, e.g., Amazon Web Services, then the resourcerequester 150 cannot provision services there. The subscription stage1220 accordingly eliminates all Amazon regions from consideration. Thesubscription analysis applies to private clouds as well. In the privatecloud scenario, the subscription information may be the credentials usedto connect to the private cloud system manager (as one example). If thecredentials are not in place in the available metadata, then thesubscription stage 1220 may consider that private cloud regionunavailable.

The placement pipeline circuitry 1216 also includes a type stage 1222that receives the current candidate set of placement options from theprevious pipeline stage (1402). The type stage 1222 determines abaseline technical component type for the resource under considerationin the technical service template 1202 (1404). For example, the typestage 1222 may identify within the technical service template 1202 theparameter fields that define a virtual machine resource type, e.g., theparameter fields for type, name, location, properties such as size, OSprofile, storage profile, and a network profile.

The type stage 1222 also receives service provider metadata 904 (1406).Given the baseline technical component type, and the service providermetadata 904, the type stage 1222 determines which service providerregions support the baseline technical component type (1408). Thisdetermination may be made in view of metadata in addition to the serviceprovider metadata 904, as well, such as the requester metadata 902 andcontainer metadata 906 that may specify particular limitations orcharacteristics of acceptable types.

In addition, the type stage 1222 is configured to initiate processing bythe TTT circuitry 116 (1410). The TTT circuitry 116 analyzes thebaseline technical component type, e.g., as described above with respectto FIGS. 2-8. The TTT circuitry 116 determines whether equivalent typesexist to the baseline technical component type, and if so, in whichregions. As such, the type stage 1222 may receive, in response to theTTT processing, additional service provider regions that support anequivalent for the baseline technical component type (1412). The typestage 1222 retains those service provider regions in the candidate setof placement options (1414) (that is, those regions are not eliminatedfrom consideration) and communicates the updated set of placementoptions to the next processing pipeline stage (1416).

Expressed another way, for every region, there is a relation defined,e.g., in database tables, between type (e.g., VM, website, or SQLdatabase) and region for that type. Not every type is available in everyregion. The type stage 1222 looks at, for the current resource theplacement pipeline circuitry 1216 is trying to place, the relationbetween the specified type that implements that resource, and theregions remaining after the subscription filter. The type stage 1222determines whether the service provider has available the specified typein that region. The type filter retains subscribed regions that supportthe specified type. In addition, the TTT processing also runs to checkwhether the specified type in the technical service template 1202 isavailable elsewhere, and whether it is available as an equivalent typein other regions.

In the example of FIG. 12, the placement pipeline circuitry 1216includes an asset stage 1224 after the type stage 1222. The asset stage1224 receives the current candidate set of placement options from theprevious pipeline stage (1502). The asset stage 1224 determines abaseline technical asset for the resource under consideration in thetechnical service template 1202 (1504). For example, the type stage 1222may identify within the technical service template 1202 the parameterfields that define an OS disk or other asset.

The asset stage 1224 also receives the service provider metadata 904(1506). Given the identified baseline technical asset, and the serviceprovider metadata 904, the asset stage 1224 determines which serviceprovider regions support the baseline technical asset (1508). Thisdetermination may be made in view of other metadata in addition to theservice provider metadata 904, such as the requester metadata 902 andcontainer metadata 906. Any of the metadata may specify particularlimitations or characteristics for acceptable assets.

In addition, the asset stage 1224 is configured to initiate processingby the TTT circuitry 116 (1510). In particular, the equivalency analysisperformed by the TTT circuitry 116 analyzes the baseline technicalasset, e.g., as described above with respect to FIGS. 2-6. Theequivalency analysis determines whether equivalent assets exist to thebaseline technical asset, and if so, in which regions. As such, theasset stage 1224 may receive, in response to the equivalency analysis,additional service provider regions that support an equivalent for thebaseline technical asset (1512). The asset stage 1224 retains thoseservice provider regions in the candidate set of placement options(1514) (rather than eliminating them) and communicates the updated setof placement options to the next processing pipeline stage (1516).

In other words, assets are associated with regions and subscriptions.Assets are referenced in the technical service template 1202 assupporting a particular resource e.g., a disk image asset. The assetstage 1224 analyzes the asset to make sure there is a relation betweenthat asset and the regions under consideration. The asset stage 1224eliminates regions that do not have a relationship with the asset. Ifthe asset (or type or network) in the technical service template is aBlue provider asset, but the resource requester does not have a Bluesubscription, then unless the asset stage 1224 finds an equivalent in,e.g., a Red provider asset, then the asset cannot be placed and thereare no placement options.

Asset metadata, e.g., reflected in the container metadata 906, mayinclude precedence information. That is, if a newer or updated versionof a particular asset (e.g., a Windows™ server disk image) is availablethen it may be used (or considered an equivalent), even if the templatespecifically calls out an older version.

The placement pipeline circuitry 1216 may also consider networks in itssearch for feasible placement options. Accordingly, the placementpipeline circuitry 1216 includes a network stage 1226 after the assetstage 1224. The network stage 1226 receives the current candidate set ofplacement options from the previous pipeline stage (1602). The networkstage 1226 determines a baseline network in the technical servicetemplate 1202 (1604). For example, the network stage 1226 may identifywithin the technical service template 1202 the parameter fields thatspecifically define a network.

The network stage 1226 may receive the service provider metadata 904(1606). Given the identified baseline network, and the service providermetadata 904, the network stage 1226 determines which service providerregions support the baseline network (1608). This determination may bemade in view of metadata in addition to the service provider metadata904, as well, such as the requester metadata 902 and container metadata906 that may specify particular limitations or characteristics foracceptable networks.

In addition, the network stage 1226 may initiate processing by the TTTcircuitry 116 (1610). In particular, the equivalency analysis performedby the TTT circuitry 116 analyzes the baseline network, e.g., asdescribed above with respect to FIGS. 2-6. The equivalency analysisdetermines whether an equivalent network exists to the baseline network,and if so, in which regions. The equivalency analysis provides theadditional regions to the network stage 1226 (1612). The network stage1226 retains those service provider regions in the candidate set ofplacement options (1614) and communicates the updated set of placementoptions to the next processing pipeline stage (1616).

Even though certain regions are otherwise feasible options for placementof a resource, those regions may not have the capacity to accept theplacement. Accordingly, in some implementations, the placement pipelinecircuitry 1216 may include a capacity stage 1228 to also considercapacity when searching for feasible placement options.

Like the prior pipeline stages, the capacity stage 1228 receives thecurrent candidate set of placement options from the previous pipelinestage (1702). The capacity stage 1228 determines the implementationrequirements for the resource under consideration (1704). For example,the capacity stage 1228 may identify within the technical servicetemplate 1202 the parameter fields that identify the number ofprocessors, amount of RAM, VM size, amount of disk space, number of VMs,number of DBs, number of application servers, amount of networkbandwidth, number of graphics processing units (GPUs), number ofencryption modules, number of network ports or interfaces, and thenumber or amount of any other components underlying the implementationof a resource.

The capacity stage 1228 receives, e.g., the service provider metadata904 (1706). Given the identified implementation requirements, and theservice provider metadata 904, the network stage 1226 determines whichservice provider regions have sufficient capacity (1708) to meet thedemands of the implementation requirements. In that regard, the capacitystage 1228 may request or receive updated service provider metadata 904to obtain an updated snapshot of current capacity. The capacity stage1228 eliminates regions from further consideration which no not have thecapacity to implement the resource (1710) and communicates the updatedset of placement options to the next processing pipeline stage (1712).

In the example of FIG. 12, the next pipeline stage is the metadata stage1230. The metadata stage 1230 receives the current candidate set ofplacement options from the previous pipeline stage (1802). The metadatastage 1230 is configured to determine which service provider regionsqualify, with regard to the particular input obtained from the resourcerequester 150, to provision the implementation of the resource underconsideration. The metadata stage 1230 thereby eliminates placementoptions responsive to the disqualified service provider regions.

In that regard, the metadata stage 1230 may receive, e.g., the requestermetadata 902 specifying particular requirements of the resourcerequester (1804), the service provider metadata 904 specifyingcapabilities of service providers (1806), and the container metadata 906specifying the properties of the technical service template (1808). Themetadata stage 1230 implements a set of metadata evaluation rules,including evaluation rules that may be resource requester specific(1810). That is, each resource requester 150 may have a customizedmetadata stage 1230 that handles issues and concerns specific to thatresource requester (as represented, e.g., within the requester metadata902) as well as issues and concerns that may be applicable acrossmultiple resource requesters.

The metadata stage 1230 executes the metadata evaluation rules todetermine whether a given service provider region passes the metadataevaluation rules (1812). Some examples are provided below. If not, themetadata stage 1230 eliminates the region from further consideration(1814). After its analysis, the metadata stage 1230 communicates theupdated set of placement options to the next processing pipeline stage(1816).

The metadata stage 1230, via the evaluation rules, analyzes resourcerequester constraints on placement. For instance, the resource requester150 may specify that certain data is subject to data security rules,e.g. for PII, PCI or ITAR data. If so, the metadata stage 1230 mayeliminate from consideration, as examples, those regions that cannotprovide the requisite level of data security, and regions that are notin an allowed geographic space (e.g., in the United States or in theEU). Note also that some of the parameters in the concretized technicalservice template derive from requester metadata 902. Accordingly, themetadata stage 1230 may also analyze the parameters in the concretizedtechnical service template and responsively make further decisions onplacement. For instance, information on required geographic placementlocations may be derived metadata in the concretized template andobtained from data originally provided as requester metadata 902.

FIG. 12 also shows a filtering and presentation pipeline stage 1232. Thefiltering and presentation pipeline stage 1232 receives the currentcandidate set of placement options from the previous pipeline stage(1902). At this final stage (for this example implementation), thecurrent candidate set of placement options have been reduced to thefeasible placement options.

The filtering and presentation pipeline stage 1232 is configured todetermining an ordering to apply (1904) and impose the ordering upon thefeasible placement options (1906). The filtering and presentationpipeline stage 1232 may also generate a GUI composing the orderedplacement options (1908) and transmit the GUI to the resource requester150 through the communication interface 202 (1910).

Note that the ordering may vary widely. In some implementations, theordering is determined by other metadata, such as the requester metadata902. For instance, the requester metadata 902 may specify a preferredordering of alphabetical order, ordering by cost, ordering by preferredservice providers, ordering by location, ordering by experience orlength of subscription, or any other mechanism for ranking the feasibleplacement options. Additionally or alternatively, the filtering andpresentation pipeline stage 1232 may impose an ordering based on serviceprovider metadata 904, including an ordering by perceived reliability,percentage of prior placement decision made to select a particularservice provider, service provider reviews or rankings, or othercriteria. The ordering may be visualized with a “star” rating, numericrating, or some other ranking indicia.

The resource requester 150 selects a placement option for each resource.Selections may be provided manually, though a GUI, or automatically,e.g., according to a pre-defined set of ordered placement preferencesfor the resource requester 150. The placement options return to the HCA112. In response, the HCA 112 may execute the TTT processing to actuallytransform the technical service template into a form suitable forexecution by service providers for the selected placement options toinstantiate the services requested by the resource requester 150. Notethat this may include splitting a baseline technical service templateinto multiple technical service templates, with the HCA 112 sending eachto the corresponding service provider hosting the selected regions. Thatis, the HCA 112 may determine which service providers host the specificresources identified in the baseline technical service template. Whendifferent service providers are involved, the HCA 112 may split thebaseline technical service template into an individual technical servicetemplate for each service provider. The HCA 112 places, in theindividual technical service templates, the resource definitions fromthe baseline technical service template for those resources that eachparticular service provider will instantiate.

The placement pipeline circuitry 1216 is a flexible, modifiable,customizable mechanism for making placement decisions. FIG. 20 showsanother example of a placement engine 2000. In this example, theplacement engine 2000 includes a cost stage 2002 added just before thefiltering and presentation pipeline stage 1232. The cost stage 2002 maydetermine the expenditure expected for placing the resources in anygiven placement location. This analysis may be done based on the figuresprovided by the service providers and represented in the serviceprovider metadata 904, for instance, for each resource, type, asset,network, or other technical component.

FIG. 21 shows another example of pipeline placement circuitry 2100. Inthis example, the placement engine 2000 includes a template modificationstage 2102 added as the first stage. The template modification stage2102 may insert, delete, or change portions of the concretized technicalservice template in response to any of the metadata in the metadatadatabase 226 according to pre-defined transformation rules. Forinstance, a transformation rule may specify that if the containermetadata 906 specifies a ‘Production’ environment, then adds a disasterrecovery section to the concretized technical service template. Thedisaster recovery section may define, for example, a set of resources toprovide automated backup for the databases defined elsewhere in theconcertized technical service template for the production environment.

FIG. 22 shows another implementation of pipeline placement circuitry2200. In this example, a second pipeline 2202 follows the first pipeline1216. The second pipeline implements the template modification stage2102, with modification done after the filtering and presentationpipeline stage 1232. The modification may implement, for instance, aresource distribution pattern by specifying implementation of resourcesto specific service providers in specific orders, e.g., round-robin, orfill-to-completion orders. There may be any number and type ofadditional pipeline stages in any number of additional pipelinesfollowing each other in sequence. In any of the placement pipelines,individual pipeline stages may be turned on or off and bypassedaccording to configuration changes made by the HCA 112.

Returning to the toy company example, the technical service template fora development environment for the toy company may define three VMs forwebservers, two VMs for application servers, and two VMs for databasesas the technical component types that makeup the developmentenvironment. The technical service template may further specify assets.Examples of the assets include a deployment package that deploys contentonto webservers and SQL databases, and OS disk images specified by imagenames for the images that provide the webserver, application server, anddatabase functionality.

As another example of how metadata influences placement, assume that themetadata database 226 establishes that the SharePoint applicationservers will be memory intensive, and need more RAM rather than diskspace or processor speed. The placement circuitry 118 may implement ametadata policy, e.g., in the type stage 1222, that memory intensiveservers preferably map to Green VMs, because the Green service providerallows much more flexibility in specifying instance types for memory.The placement circuitry 118 may responsively map the two applicationservers away from the Blue VMs to the Green VMs, as long as TTTcircuitry 116 has established a type mapping from Blue to Green. As aresult of such a mapping defined by the TTT databases, the SharePointprovisioning may result in three Blue VMs for web front ends, two GreenVMs for application servers, and two Red VMs for the data tier.

Re-Placement

The HCA 112 described above supports dynamic re-determination ofplacement options and initiating re-placement of resources specified inthe technical service templates 908. As one example, the HCA 112 mayreceive specific re-placement requests from the resource requester 150,and in response, re-evaluate the feasible placement options forindividual resources or sets of resources, e.g., those defined in atechnical service template. To do so, the HCA 112 may re-execute theplacement pipeline circuitry to determine whether there are any updatedplacement options that specify a new possible placement location for anyof the resources. In connection with the re-evaluation, the HCA 112 mayobtain updated container metadata 906, requester metadata 902, serviceprovider metadata 904, or any other input data prior to re-executing theplacement pipeline circuitry.

FIG. 23 shows an example of a HCA 2300 that supports dynamicre-placement. In the HCA 2300, the system circuitry 204 is extended toinclude re-evaluation circuitry 2303 for dynamic re-determination ofplacement options and initiating re-placement of resources. Forinstance, the processor 216 may execute dynamic placement instructions238, described in more detail below, for dynamic re-determination andre-placement.

The HCA 2300 also includes extended metadata 2304. In particular, theextended metadata 2304 includes re-placement metadata 2306. There-placement metadata 2306 may specify re-evaluation timing 2308,re-evaluation trigger conditions 2310, maintenance or update windows2312, or other re-placement variables. These variables may be set by anindividual responsible for setting up the re-evaluation properties forany given resource request, or, for example, a pre-defined set ofbaseline re-evaluation properties may be inserted as the re-placementmetadata 2306. The HCA 2300 may attach the re-placement metadata 2306 totechnical service templates, resources, assets, types, service requests,resource requesters, service providers, or at other granularities.

Expressed another way, the re-placement metadata 2306 may be attached tothe technical service templates 908 as a whole, or to individualcomponents within the technical service templates 908. That is, the HCA2300 may define a link between the re-placement metadata 2306 and thetechnical service template, between the re-placement metadata 2306 andindividual resources in the technical service template, or at anotherlevel of granularity. The link may be, e.g., a database record that tiesall or part of the re-placement metadata 2306 to another object, such asthe technical service template. As examples, the re-placement metadata2306 may be linked to resources, assets, types, or other individualcomponents defined within the technical service templates 908. There-placement metadata 2306 may also extend or link to any other metadatain the HCA 2300, such as the requester metadata 902, container metadata906, and service provider metadata 904.

FIG. 24 shows an example of dynamic re-placement 2400, and FIG. 25 showsa corresponding logical flow 2500. In this example, the re-evaluationcircuitry 2302 determines re-evaluation timing 2402 attached to a VMresource, VMa 2404 (2502) and determines a re-evaluation trigger 2406attached to the VMa 2404 (2504). The re-evaluation circuitry 2302 maydetermine these parameters by reading the re-placement metadata 2306linked to the VM resource. The re-evaluation timing 2402 specifies whento perform re-evaluation. As just a few examples, the re-evaluationtiming 2402 may specify a regular interval, e.g., nightly, or weekly; aspecific date, e.g., on June 25th; a specific time, e.g., at 11 pm eachnight; or some other time or date specifier or combination of time anddate. The re-evaluation trigger specifies a particular condition thatwill cause re-evaluation regardless of the re-evaluation timing. As oneexample, the re-evaluation trigger may be a metadata update, forinstance, a service provider update of service provider metadata 904.Another example of a re-evaluation trigger is a request forre-evaluation received from the resource requester 150. Another exampleis an update made to a technical service template used to define orprovision resources for the resource requester 150.

The re-evaluation circuitry 2302 determines when to initiatere-evaluation in response to the re-evaluation timing and there-evaluation triggers (2506). That is, when the re-evaluation timing ismet, or the re-evaluation trigger fires, the re-evaluation circuitry2302 initiates re-evaluation of the resource, preferably using thecurrent updated metadata and technical service template (2508). FIG. 24shows three example instances of re-evaluation: timing instance 2408,trigger instance 2410, and timing instance 2412.

Initiating re-evaluation may include providing the technical servicetemplate 908, current metadata in the metadata database 226, andidentification of the resource (VMa 2404) to the placement circuitry 118(2510). The re-evaluation circuitry 2302 receives in response an updatedset of placement options (2512), e.g., the updated placement options2414, 2416, and 2418 in the example of FIG. 24. The re-evaluationcircuitry 2302 also determines whether the updated set includes any newplacement locations for the resource, e.g., the new placement option2420 for the VMa 2404 (2514).

If new locations are available for placing the resource (2516), then there-evaluation circuitry 2302 may determine whether to actually initiatethe re-placement (2518), and if multiple new locations are possible, theselected new location. For instance, the re-evaluation circuitry 2302may send a re-placement authorization request to the resource requester150 and receive an acceptance response or denial response. As anotherexample, the re-evaluation circuitry 2302 may automatically determinewhether to re-place the resource by evaluating a pre-definedre-placement rule. Examples of a re-placement rule include: alwaysperform re-placement; perform re-placement if the resource belongs tospecific resource requesters; perform re-placement if the new locationis with a preferred service provider; perform re-placement if the newlocation is a preferred location; and perform re-placement if theexpected cost saving for hosting the resource at the new locationexceeds a cost threshold. As an example, assume that VMa, whichimplements a data server, is initially placed in the Blue serviceprovider region U.S. West. After the initial placement, the Red serviceprovider implements a higher-speed VM resource connected to higher-speednetworks. The re-placement process may move VMa from the Blue serviceprovider region to the Red service provider region to take advantage ofthe faster VM and network connectivity.

Re-placement may be accomplished in different ways. For instance, when adecision is made to re-place the resource, the re-evaluation circuitry2302 may initiate instantiation and provisioning of a replacementresource first, at the selected new location (2520). The re-evaluationcircuitry 2302 read the re-evaluation metadata to determine amaintenance window for the resource requester (2522). FIG. 24 showsexample maintenance window metadata 2422 defining a maintenance window2424. The maintenance window defines, for the particular resourcerequester, when service may occur on its resources. When the window isopen, the re-evaluation circuitry 2302 initiates switch-over to the newresource (2524), and shut down of the existing resource. Switch-over mayinclude pausing the existing resource, and copying the current state ofthe resource to the newly instantiated replacement. FIG. 24 showsswitchover 2426 to the new resource, VMb 2428. The switch-over may beaccomplished by initiating an update of routing tables to point to thenew resource, after the existing machine state is replicated at the newresource, for instance.

Expressed another way, the re-evaluation circuitry 2302 determines andtakes action on maintenance windows attached to resources. Each resourcein the HCA 112 may have re-placement metadata attached to it (e.g.,through a database table link) that defines the maintenance window whenthe resource requester will accept some amount of outage or downtime to,e.g., move resources. The re-evaluation circuitry 2302 wait until thewindow opens for switch-over to avoid major interruptions. Themaintenance window may be part of the requester metadata 902 collectedfrom the resource requester 150. Re-evaluation may be performed on anybasis, including timing and triggers defined in the re-placementmetadata 2306. As one example, the re-evaluation circuitry 2302 mayevaluate, for example, every resource in every workload every week andreturn recommendations to each resource requester.

If moving the resource is authorized, then re-placement is performed,with actual switch-over occurring, e.g., in the migration window. Thatis, the re-evaluation circuitry 2302 may setup the switch beforehand byprovisioning new resources in a new region ahead of time, because insome cases significant time is needed to setup the replacement resource.Once the new resources are provisioned, the actual switch to the newresource may wait until the migration window is open. Alternatively, there-evaluation circuitry 2302 may perform an offline migration during themaintenance window by shutting down the resource, copying over to thenew location, and restarting the resource.

FIG. 26 shows an example of offline dynamic re-placement 2600,continuing the example shown in FIG. 24. In FIG. 26, the updatedplacement decisions 2602, 2604, and 2606 result from re-evaluatingplacement options. The updated placement decisions 2602 include a newplacement location 2608. When the maintenance window 2610 opens, there-evaluation circuitry 2302 initiates shut-down 2612 of the resource(the VMa 2404 in this example). The re-evaluation circuitry 2302 thencopies 2614 the resource to the new placement location 2608. The copiedresource reboots 2616, and the re-evaluation circuitry 2302 initiatesswitch-over 2618, e.g., by updating routing tables. The re-placementneed not complete within the maintenance window 2610.

The HCA 2300 includes placement pipeline circuitry comprising multipleprocessing stages configured to determine initial placement options fora technical component (e.g., a type like a VM, or assets like OS disks)of a specified service request. The HCA 2300 stores (e.g., asre-placement metadata 2306), timing metadata linked to the technicalcomponent. The timing metadata defines a dynamic re-evaluation timingspecifier for re-evaluating placement of the technical component. There-evaluation circuitry 2302 is responsive to the dynamic re-evaluationtiming specifier to re-execute the placement pipeline circuitry on thetechnical component and determine updated placement options including anew placement location for the technical component.

Note that the specified service request is linked to a specific resourcerequester. Placement execution metadata for the specific resourcerequester defines an update time window (e.g., a maintenance window) formaking adjustments to the specified service request. The re-evaluationcircuitry 2302 initiates instantiation of a replacement component forthe technical component at the new placement location responsive todetermining the updated placement options. Further, the re-evaluationcircuitry initiates switchover to the replacement component within theupdate time window.

Several examples follow of changes that may cause new placementlocations to become available. The placement pipeline circuitry 1216includes a subscription stage 1220 that may determine to a change towhich service provider regions the resource requester has activesubscriptions, and thereby determine new placement locations. Theplacement pipeline circuitry 1216 also includes a type stage 1222 and anasset stage 1224 configured to determine a change to which serviceprovider regions support the technical components, and thereby determinethe new placement locations. Similarly, the capacity stage 1228 maydetermine a change in which service provider regions have capacity toprovision the technical component, and thereby determine the newplacement locations. In addition, the metadata stage 1230 may determinea change to which service provider regions qualify to provision thetechnical component and thereby determine the new placement locations.

The HCA 2300 receives a technical service template for implementing aservice request for a resource requester. The HCA 2300 identifies aresource (e.g., a VM) within the technical service template, andexecutes, for the resource, placement pipeline circuitry comprisingmultiple processing stages configured to determine initial placementoptions for the resource. The HCA 2300 also executes re-evaluationcircuitry 2302 configured to determine when to re-execute the placementpipeline circuitry for the resource and determine updated placementoptions including a new placement location for the resource.

Timing metadata linked to the resource provides a timing specifier forre-evaluating placement of the resource. The HCA 2300 also obtainsplacement execution metadata linked to the resource requester. Theplacement execution metadata defining an update time window forimplementing the new placement location. Accordingly, the HCA 2300 mayinitiate provisioning of a replacement for the resource at the newplacement location responsive to determining the updated placementoptions and initiate switchover to the replacement within the updatetime window.

Placement and Provisioning Architecture

FIG. 27 shows a cloud computing placement and provisioning architecture2700. This particular example of the cloud computing placement andprovisioning architecture 2700 is grouped for purposes of illustrationinto placement circuitry 2750 and provisioning circuitry 2752. Otherimplementations may vary widely. FIGS. 28 and 29 show correspondinglogical flows 2800 and 2900, respectively. The HCA 112 generates a GUI2702, e.g., rendered locally at the resource requester 150. The GUI 2702displays, in this example, the services available to the resourcerequester 150 as reflected in a service catalog 2704 maintained for theresource requester 150 (2802). The resource requester 150 issues atechnical service request (TSR) 2706 for virtualized hosting, e.g., forthe toy company development environment. The HCA 112 receives thetechnical service request (TSR) (2804) 2706 and initiates end-to-endplacement and provisioning analysis and actions (2806), examples ofwhich are shown in FIG. 27.

For instance, the HCA 112 may retrieve the baseline technical servicetemplate for the development environment, and the requester metadata902, container metadata 906, and service provider metadata 904 (2808).The HCA 112 provides these inputs to the placement circuitry 118 (2810),which determines the placement options 2708 for each resource in thebaseline technical service template (2812). If there are no placementoptions for a particular resource, then it may not be possible toprovision the development environment. However, if each resource has aplacement option, then the HCA 112 may request the resource requester150 to make placement decisions 2710 (2814).

The TTT circuitry 116 transforms the baseline technical service templateto meet the technical component specification details expected by theregions where the resources will be placed (2816). As discussed above,the TTT circuitry 116 may perform equivalency analysis to findequivalent assets and may also perform type translation to identify andspecify equivalent types (e.g., VMs). When the service request willprovision resources to multiple different regions or service providers,then the TTT circuitry 116 may also split the baseline technical servicetemplate into multiple individual templates 2712, each specifyingresources for a particular servicer provider or region (2818).

A first dispatcher 2714 receives the templates (2820) and decides,responsive to, e.g., the service provider or region, which provisioningsystem should receive the template (2822). That is, the HCA 112 mayhandoff a template to an external service provider system (2824), e.g.,a Microsoft™ Azure™ stack. FIG. 27 shows an external service provider2716 that receives a particular template. In response to receiving theparticular template, the external service provider 2716 performs theprovisioning actions that lead to instantiation of the resourcesspecified in the template in the Black region 2718.

The HCA 112 may process templates by passing them to the job preparationcircuitry 2720, which may be referred to as a job manager or shredder(FIG. 29, 2902). The HCA 112 executes the job preparation circuitry 2720to prepare new jobs and tasks that implement the jobs for provisioningthe resources in the template (2904). The job preparation circuitry 2720stores the new provisioning job and the tasks in a pending job database2722 (2906). These represent pending provisioning jobs implemented withpending tasks. In this regard, the job preparation circuitry 2720 readsthe dependencies in the template, and specifies tasks in the reverseorder to satisfy the dependencies. For instance, when the templatespecifies that a development environment relies on a web tier and a datatier for operation, the job preparation circuitry 2720 may create adevelopment environment job, including a task to provision the web tierfirst, then a task to provision the data tier, then a task toinstantiate the development environment itself.

The polling circuitry 2724, on a pre-determined schedule, queries thejob preparation circuitry 2720 for pending tasks (2908). The pollingcircuitry 2724 may continue to query for new provisioning jobs as longas the polling circuitry 2724 remains running. When a new provisioningjob is found, the polling circuitry 2724 obtains the underlying pendingtasks in the order specified for implementation by the job preparationcircuitry 2720 (2910).

The polling circuitry passes the pending tasks to the dispatchercircuitry 2726 (2912). The dispatcher circuitry 2726 decides to whichworkflow to send the pending tasks (2914), and sends the pending tasksfor execution (2916). The workflows are defined, e.g., by runbooks, inthe provisioning workflow circuitry 2728. The runbooks may beimplemented as a pre-defined set of procedures and operations carriedout by a system to accomplish a task. The provisioning workflowcircuitry 2728 may execute service management automation (SMA) or othertools for executing any pending task, e.g., via by calling a selectedrunbook for that task (2918). The provisioning workflow circuitry 2728communicates with the service providers, responsive to the provisioningactions carried out under direction of the runbooks. As a result, theresources specified in the templates, and the resources that constitutethe requested development environment, become provisioned in any numberof service provider regions, e.g., the Red region 2730, the Green region2732, and the Blue region 2734.

FIG. 30 shows another example of the cloud computing placement andprovisioning architecture 3000, including placement circuitry 3002 andprovisioning circuitry 3004. In this example, the resource requester 150has requested a development environment. The baseline template 3006defines that the development environment includes resourcescorresponding to four VMs, two of size 1 for the data tier, one of size2 for the application tier, and one of size 3 for the web front end.

The requester metadata 3008 specifies PCI data security, applicable tothe data tier. The service provider metadata 3010 specifies that: theRed region supports PCI data, and size 1 and 2; the Green region doesnot support PCI data, and supports size 1 and 2 VMs; and the Blue regiondoes not support PCI data, and supports size 2 and 3 VMs.

Responsive to the baseline template 3006, service provider metadata3010, and requester metadata 3008, the placement circuitry 118determines placement options for each resource in the baseline template3006. In this scenario, the placement circuitry 118 determines that bothVMs in the data tier must be provisioned in the Red region, because onlythe Red region supports PCI data for the size 1 VMs that constitute thedata tier. The placement circuitry 118 also determines that the size 2VM for the application tier may be placed in any of the Red, Green, orBlue regions. In this example, the resource requester chooses the Greenregion. Finally, the placement circuitry determines that only the Blueregion can host the size 3 VM for the web front end.

In other words, the placement circuitry 3002 has found a way to locatethe set of resources needed for the development environment within thelarge search space of multiple different providers and regions. Theprovisioning circuitry 3004 may then coordinate instantiation of thesize 1 VMs for the data tier into the Red region 2730, the size 2 VM forthe application tier into the Green region 2732, and the size 3 VM forthe web front end into the Blue region 2734. To that end, the placementcircuitry 3002 may split the baseline template into multiple, e.g., 3,concretized technical service templates, one for each region. Theplacement circuitry 3002 passes each concretized technical servicetemplate to the job preparation circuitry 2720 for processing.

The example above addressed VMs and data security requirements. However,as noted above, the placement circuitry 3002 may address other technicalcomponent types as well, in addition to different types of assets, suchdisk images.

Expressed another way, the HCA 112 includes a communication interfaceconfigured to receive a selection of a computing environment (e.g., adevelopment environment) for provisioning from a resource requester. TheHCA 112 also includes placement circuitry in communication with thecommunication interface that determines placement options (e.g., the Redregion, Blue region, or Green region) for a resource type (e.g., a GreenVM) for implementing part of the computing environment. The placementcircuitry 118 also obtains from the resource requester 150 a selectedplacement chosen from among the placement options. TTT circuitry 116 inthe HCA 112 determines a service provider region corresponding to theselected placement and translates the resource type to a destinationtype (e.g., a Blue VM) for provisioning in the service provider region.Provisioning circuitry 2752 initiates provisioning of the destinationtype within the service provider region. The provisioning circuitry 2752may vary widely in implementation, for instance including the jobpreparation circuitry 2720, polling circuitry 2724, dispatcher circuitry2726, and provisioning workflow circuitry 2728. Other implementations ofthe provisioning circuitry 2752 may include the additional dispatcher2714, or have additional or different circuitry.

As noted above, for determining the placement options, the placementcircuitry 118 may receive a technical service template for the computingenvironment, with the technical service template specifying the resourcetype. The placement circuitry 118 may also receive container metadatacharacterizing a structural organization of the resource requester,requester metadata specifying implementation options of the resourcerequester for the computing environment, and service provider metadataspecifying available technical components available from differentservice providers.

The job preparation circuitry 2720 prepares a new job and tasks thatimplement the new job for provisioning the destination type. The jobpreparation circuitry 2720 stores the new job and the tasks in a pendingjob database as pending jobs with pending tasks. The polling circuitry2724 is configured to query the job preparation circuitry 2720 for thepending jobs with the pending tasks. As explained above, the dispatchercircuitry 2726 obtains the pending tasks and provides the pending tasksto the provisioning workflow circuitry 2728. The provisioning workflowcircuitry 2728 initiates provisioning of the destination type within theservice provider region by sending the pending tasks to a serviceprovider system responsible for instantiating resources in the serviceprovider region.

FIG. 31 shows an example 3100 of a baseline technical service template3102 and a concretized technical service template 3104. The baselinetechnical service template 3102 identifies resource types, parametersapplicable to the resource types, placement specifications for theresource types, and optionally other parameters that describe andcharacterize a resource or set of resources supported by a serviceprovider. In this example, the baseline service template 3102 definestwo resources: a VM resource 3106 and a website resource 3108. Note thatthe baseline technical service template 3102 specifies that a VMresource 3106 is located in the West US region (the placementspecification for the resource), and has a size parameter 3110 andinstance parameter 3112 (the parameters that describe the VM resource).The website resource 3108 parameterizes its location via the regionparameter 3114. Turning ahead briefly, FIG. 38 shows one of manyexamples of coding 3800 to define a VM resource in a baseline technicalservice template.

Returning to FIG. 31, the concretized technical service template 3104specifies values for the parameters in the baseline technical servicetemplate 3102. In this example, the size of the VM resource is ‘A4’,which implies specific technical features to the service provider, e.g.,number of processors and RAM, while the number of instances has been setto ‘2’. In the concretized technical service template 3104, the regionparameter for the web site resource 3108 has been set to ‘West US’.

FIG. 32 provides an illustration of service provider metadata 3200defining types, networks, and assets for the Blue service provider 3202.In this example, the service provider metadata 3200 describes the WestUS region 3204. The service provider metadata 3200 establishes that theWest US region 3204 supports: VM resources 3206 and storage resources3208; a corporate network 3210 and an open network 3212; and assetsincluding a 2012 server disk image 3214 and a 2014 server disk image3216.

FIG. 33 shows an example of a region roll-up 3300. In this example, thetop level West region 3302 defines two sub-region roll-ups, the primaryregion roll-up 3304 and the secondary region roll-up 3306. The primaryregion roll-up 3304 includes the Red provider ‘West US’ region 3308 andthe Blue provider ‘Pacific NW’ region 3310. The secondary region roll-up3306 includes the Red provider ‘Northwest’ region 3312 and the Greenprovider ‘California’ region 3314. With regard to the placementcircuitry 118 and the TTT circuitry 116, regions under a common roll-up(e.g., the Northwest region 3312 and the California region 3314) aretreated as equivalent regions, and the placement circuitry 118 and theTTT circuitry 116 may substitute one for another in terms of findingfeasible placement locations.

FIG. 34 shows an example network roll-up 3400. In this example, thenetwork roll-up 3400 has defined three networks under the top levelnetwork ‘Corpnet’ 3402. The network equivalence processing discussedabove with regard to the network pipeline stage 1226 may consider any ofthe Red West US network 3404, the Blue California network 3406, or theBlue Seattle network 3408 equivalent by virtue of their membership underthe Corpnet network 3402. Accordingly, even if the specificallydesignated network, e.g., the Red West US network 3404, is notavailable, then placement options include those regions where any of theother networks are available, i.e., the Blue California network 3406 orthe Blue Seattle network 3408.

FIG. 35 further illustrates network equivalence 3500. In the example3502, the baseline service template 3504 specifies ‘Vnet x as thenetwork. The TTT circuitry 116 parses the network roll-up 3400 to findthat any of the Blue California network 3406, or the Blue Seattlenetwork 3408 are equivalent and may be substituted, due to theirinclusion under the Corpnet 3402 abstraction with ‘Vnet x’ 3404. In theexample 3506, the baseline service template 3508 specifies ‘vlan 432’ asthe network. Similarly, the TTT circuitry 116 parses the network roll-up3400 to find that both the Red West US network 3404 and the BlueCalifornia network 3406 are equivalent and may be substituted.

FIG. 36 shows an example asset roll-up 3600. In this example, the assetroll-up 3600 has defined, as the top level asset, the disk image for theServer 2012 operating system 3602. The asset roll-up 3600 includes fouralternative disk images: Red region, /path/s2012R2.vhd 3604; Red region,/path/s2012R3.vhd 3606; Blue region/path/server2012.ami 3608; and Greenregion/path/srv2012R2.vhd 3610. The asset equivalence processingdiscussed above considers a designation of the Server 2012 R2 Base OSImage 3602 satisfied by (e.g., equivalent to) the images 3604-3610.

FIG. 37 further illustrates asset equivalence 3700. In the example 3702,the baseline service template 3704 specifies Server 2012 R2 Base OSImage 3602 as the baseline disk image. The TTT circuitry 116 parses theasset roll-up 3600 to find that any of the alternative disk images: Redregion, /path/s2012R2.vhd 3604; Red region, /path/s2012R3.vhd 3606; Blueregion/path/server2012.ami 3608; and Green region/path/srv2012R2.vhd3610 are equivalent and may be substituted. In the example 3706, thebaseline service template 3708 specifies the disk image Red region,/path/s2012R2.vhd 3604. Similarly, the TTT circuitry 116 parses theasset roll-up 3600 to find that Red region, /path/s2012R3.vhd 3606; Blueregion /path/server2012.ami 3608; and Green region/path/srv2012R2.vhd3610 are equivalent and may be substituted.

Provisioning Architecture with Template Aggregation

FIG. 41 shows a cloud resource provisioning architecture 4100(architecture 4100) with template aggregation. This particular exampleof the architecture 4100 is divided, for purposes of explanation, intoplacement circuitry 4102 and provisioning circuitry 4104. Otherimplementations may vary widely. FIGS. 42, 43, and 44 show correspondinglogical flows 4200, 4300, and 4400 that the architecture 4100 mayimplement.

The placement circuitry 4102 may operate as described above with respectto FIGS. 27, 28, 29, and 30, for example. For instance, the HCA 112receives a technical service request (TSR) 2706 (4202) and initiates aplacement analysis and provisioning (4204) for the resources in thecorresponding baseline technical service template 4106 for the technicalresource request 2706. As noted above, the placement analysis mayinclude determining the placement options for the resources specified inthe template, obtaining placement decisions for service provider regionsfrom the resource requester 150, and executing TTT circuitry 116 totransform baseline resources to the form expected by the selectedservice provider regions (4206).

In this example, the TSR 2706 is for a development environment, and forthe purposes of discussion below the corresponding baseline template4106 specifies an instance of a Machine Learning (ML) service, and fourVMs: Two of size 1 and two of size 2. It is also assumed that (whetherthrough placement options selected by the resource requester 150, or dueto other constraints) the two size 1 VMs will be placed in the Redservice provider region 2730 (a public region), the two size 2 VMs willbe placed in the Black service provider region 4108 (an on-premisesregion), and the ML instance will be placed in the Blue service providerregion 2734 (a public region).

The provisioning circuitry 4104 in the architecture 4100 includestemplate dispatcher circuitry 4108, job preparation circuitry 4110, andresource correlation circuitry 4112. The job preparation circuitry 4110communicates with a source of templates, such as the template database4114. The templates follow a predefined format, for instance, the formatof Azure Resource Management (ARM) templates, and specify the set ofresources needed to instantiate the TSR 2706.

The resource correlation circuitry 4112 facilitates provisioning ofresources to both public clouds and on-premises clouds. To that end, theprovisioning circuitry 4104 includes a public cloud queue 4116,on-premises (private) cloud queues 4118, and a return queue 4120. Inaddition, the provisioning circuitry 4104 includes a public cloudprovisioning workflow engine 4122, which is in communication with asource of provisioning scripts, such as the public cloud script database4124.

As will be explained in more detail below, the resource correlationcircuitry 4112 may issue resource queries 4126 to a source of resourceinformation, such as the TTT circuitry 116. As examples, the resourcequeries may be made through a request interface, such as the correlationAPI 4128, or may be made as database management system queries. Therequest interface returns resource characteristics 4128 to the resourcecorrelation circuitry 4112. Examples of resource characteristics 4128include: the service provider region in which the resource will beplaced, whether the resource may be aggregated together with otherresources for provisioning, whether the resource is template deployable,and a script locator (e.g., a uniform resource indicator (URI)) thatspecifies a script that handles provisioning of the resource. Thetemplate deployability characteristic may specify whether the serviceprovider region has the ability to natively deploy the resource, given aresource template that specifies the resource in the format defined bythe specific service provider region. Not all resources are templatedeployable by the service provider region; in this example, the MLinstance is not template deployable. Non-template deployable resourcesmay be instantiated by calling specific pre-defined API functionsexposed by the service provider region, to cause the service providerregion to take the specific deployment actions needed to instantiate theresource.

FIG. 41 also shows an on-premises service provider 4130. The on-premisesservice provider 4130 locally handles provisioning of resources in theprivate on-premises environment. In support of local provisioning, theon-premises service provider 4130 includes a provisioning workflowengine 4132. The provisioning workflow engine 4132 is in communicationwith a source of provisioning scripts, such as the on-premises scriptdatabase 4134. There may be any number of on-premises service providers4130, and each may be in communication with a private cloud queuespecific to that on-premises service provider. That is, the privatecloud queues 4118 may define tenant specific queues for the on-premisesservice providers.

The job preparation circuitry 4110 is in communication with the templatedispatcher circuitry 4108 and receives a provisioning request messagefor a system deployment (e.g., for the TSR 2706) from the templatedispatcher circuitry 4108 (4208). The provisioning request message maybe, or may include, a template identifier URI. The job preparationcircuitry 4110 obtains the template identifier (4210), and retrieves aprovisioning template specified by the template identifier forimplementing the system deployment (4212). In that regard, for example,the URI may point to a specific provisioning template in the templatedatabase 4114.

To continue the example of the deployment of the developmentenvironment, FIG. 42 shows that the provisioning template 4214 specifiesthe set of resources included with the development environment. That is,the provisioning template 4214 includes resource specification sections4216 that enumerate each of the resources in the developmentenvironment. In this case, the resource specification sections 4216specify the characteristics for two VMs of size 1, two VMs of size 2,and instance of ML. The provisioning template 4214 may include anynumber and type of resources.

The job preparation circuitry 4110 disaggregates the resources in theprovisioning template 4214 into separate resource provisioning tasks4218, 4220, 4222, 4224, and 4226 for corresponding disaggregatedresources (4220). That is, the job preparation circuitry 4110 prepares aseparate provisioning task for each resource in the provisioningtemplate 4214. The job preparation circuitry 4110 may assign correlationidentifiers to each of the separate provisioning tasks 4218-4226. Thecorrelation identifiers may identify the separate provisioning 4218-4226tasks as belonging to the instantiation of the system requested by theTSR 2706.

The resource correlation circuitry 4112 communicates with the jobpreparation circuitry 4110, and determines each of the disaggregatedresources for instantiating the requested system. In one implementation,the resource correlation circuitry 4112 receives and analyzes theseparate provisioning tasks, or otherwise obtains an identification ofeach resource involved in the system deployment (FIG. 43, 4302). In thedevelopment environment example, the disaggregated resources are: 1) VMSize 1, 2) VM Size 1, 3) VM Size 2, 4) VM Size 2, and 5) ML.

The resource correlation circuitry 4112 queries the correlation datarequest interface (e.g., the correlation API 4128) to determinecharacteristics of each resource (4304). The resource characteristicsprovide information from which the resource correlation circuitry 4112determines whether to aggregate resources. As examples, thecharacteristics may include the resource provider region for deploymentof the resource, and whether the disaggregated resource may beaggregated with other resources for deployment. As another example, thecharacteristics may include the resource provider region for deploymentof the resource, and whether the disaggregated resource is templatedeployable. The characteristics may also include a provisioning scriptidentifier for each disaggregated resource for executing theprovisioning steps for the resource.

As will be described further below, the resource correlation circuitry4112 may communicate the provisioning script identifier to the publiccloud queue 4116 for processing by the public cloud provisioningworkflow engine 4122, or to the private cloud queue 4118 for processingby the on-premises provisioning workflow engine 4132. The provisioningscript identifier may be or may specify a resource locator (e.g., a URI)for a provisioning script in a script repository in communication withthe provisioning workflow circuitry.

For the development environment example, the resource characteristics4306 that the correlation API 4128 returns to the resource correlationcircuitry 4112 are:

1) VM Size 1: Red Region, Aggregate=True;

2) VM Size 1: Red Region, Aggregate=True;

3) VM Size 2: Black Region, Aggregate=True;

4) VM Size 2: Black Region, Aggregate=True; and

5) ML: Blue Region, Aggregate=False.

The resource correlation circuitry 4112 determines correlated resourcesamong the disaggregated resources (4308). The resource correlationcircuitry 4112 may apply any pre-defined correlation test to make thisdetermination. For instance, the correlation test may be thatdisaggregated resources are correlated resources when they will beplaced in a common resource provider region, and when each of thedisaggregated resources is template deployable in the common resourceprovider region. As another example, the correlation test may be thatdisaggregated resources are correlated resources when they will beplaced in a common resource provider region, and when the resourcecharacteristics directly specify that the resource may be aggregatedtogether.

For the development environment example, the two VMs of Size 1 arecorrelated. Also, the two VMs of Size 2 are correlated. The ML instanceis un-correlated with any other resource because it has been flagged bythe TTT circuitry 116 as a resource that cannot be aggregated, e.g.,because that resource is not template deployable. That is, the resourcecorrelation circuitry 4112 also determines the un-correlated resourcesamong the disaggregated resources (4310).

The resource correlation circuitry 4112 aggregates sets of correlatedresources into common resource provisioning template blocks (4314). Forthe development environment example, the two VMs of Size 1 areaggregated into a template block 4316 and the two VMs of Size 2 areaggregated into the template block 4318. The ML instance remainsseparate. The template blocks 4312 and 4314 may be single files, datastructures, or other composite data entities that include resourcecharacteristics and provisioning data for each resource aggregated intothe template block.

The resource correlation circuitry 4112 submits the common resourceprovisioning template blocks to the provisioning workflow circuitrytasked with facilitating provisioning of the correlated resources(4320). For the development environment example, the resourcecorrelation circuitry 4112 submits the VM Size 1 template block 4316 tothe public cloud queue 4116, because these VMs will be placed in the Redservice provider region, which is a public cloud region. Forun-correlated resources, for which there is no template block, theresource correlation circuitry 4112 may submit a separate provisioningmessage to the provisioning workflow circuitry. For the developmentenvironment example, the resource correlation circuitry 4112 submits aprovisioning message to the public cloud queue 4116 for the ML instance,for placement in the Blue public cloud service provider region.

Because the template block 4318 specifies resources for the Blackon-premises service provider region, the resource correlation circuitry4112 submits the template block 4318 to the private cloud queue 4118.Both the submission of a template block for correlated resources and thesubmission of a provisioning message for an un-correlated resource mayinclude a provisioning script identifier. The provisioning scriptidentifier may be, e.g., a URI into the script database 4124 or thescript database 4134 that locates the particular provisioning script torun to cause deployment actions for the resource or template bock ofresources.

The public cloud provisioning workflow engine 4122 and the on-premisesprovisioning workflow engine 4132 facilitate deployment of resources.The public cloud provisioning workflow engine 4122 may be implemented,as one example, as a C# .net component in an Azure web job, with anattached schedule. The on-premises provisioning workflow engine 4132 maybe implemented as a Windows™ service running under Windows™ server, alsooperating under a schedule.

The public cloud provisioning workflow engine 4122 checks the publiccloud queue 4116 according to the attached schedule (FIG. 44, 4402), andretrieves messages specifying new tasks to execute (4404). Similarly,the on-premises provisioning workflow engine 4132 connects to its tenantspecific on-premises cloud queue 4118 to check for and retrieve messagesspecifying new tasks (4402, 4404). In that regard, the on-premisesprovisioning workflow engine 4132 is configured with the accesscredentials (e.g., login ID and password and encryption keys) to accessits tenant specific queue.

As explained above, the tasks may be directed to deployment of multipleresources within a template block or to the deployment an individualresource. Each of the messages specifying a task may include a scriptidentifier for a script that the particular provisioning workflow enginewill execute to provision the resources. FIG. 44 shows an example scriptidentifier 4406 attached to the template block 4316, a script identifier4408 attached to the provisioning message 4312, and a script identifier4410 attached to the template block 4318. The provisioning workflowengines 4122 and 4132 obtain the provisioning script (e.g., from thescript database 4124 and 4134) and execute the script (4412).

In the example implementation discussed above, one condition forcreating the template blocks was that the individual resources in thetemplate block were template provisionable by the host service providerregion. As such, the provisioning script for template blocks may be apass-through execution instruction 4414 that forwards the template blockto the resource provider region, with a provisioning request orinstruction to the resource provider region to natively instantiate thecorrelated resources specified in the template block. The provisioningscript for uncorrelated resources, however, may call service providersinterfaces (e.g., the Blue interface API 4416) to invoke the specificfunctions made available by the service provider to instantiateresources.

The on-premises provisioning workflow engine 4132 and the publicprovisioning workflow engine 4122 save, in the return queue 4120,provisioning result messages 4420 that specify completions, failures,error conditions, or any other result information for the resourceprovisioning actions (4418). The return queue 4120 makes the resultmessages available for tracking status of resource provisioning requestsback through the provisioning chain, to the resource correlationcircuitry 4112, job preparation circuitry 4110, and dispatcher circuitry4108. In that regard, the result messages may include a correlationidentifier that identifies to which resources the result messages apply.

Expressed another way: a cloud resource provisioning architecture withtemplate aggregation includes template dispatcher circuitry configuredto prepare a provisioning request message. The provisioning requestmessage includes a template identifier of a provisioning template thatspecifies implementation of a first resource and a second resource. Thetemplate dispatcher circuitry submits the provisioning request messageto job preparation circuitry to initiate provisioning of the firstresource and second resource.

Job preparation circuitry in communication with the template dispatchercircuitry receives the provisioning request message from the templatedispatcher circuitry and obtains the template identifier from theprovisioning request message. The job preparation circuitry alsoretrieves the provisioning template specified by the templateidentifier, the provisioning template specifying implementation for boththe first resource and the second resource. The job preparationcircuitry disaggregates the first resource from the second resource, bypreparing separate resource provisioning tasks for the first resourceand the second resource.

Resource correlation circuitry in communication with the job preparationcircuitry queries a resource service (e.g., the correlation API 4128) onthe first resource and obtains a first service provider regionidentifier and a first aggregation indicator. The resource correlationcircuitry also queries the resource service on the second resource andobtains a second service provider region identifier and a secondaggregation indicator. The resource correlation circuitry determinesthat the first service provider region identifier and the second serviceprovider region identifier both identify a common service providerregion, determines that the first aggregation indicator is True,determines that the second aggregation indicator is True, and thenaggregates the first resource and the second resource into a commonresource provisioning template block. The resource correlation circuitryalso submits the common resource provisioning template block to workflowprovisioning circuitry tasked with facilitating provisioning of thecorrelated resources.

The cloud resource provisioning architecture also includestenant-specific queues for on-premises cloud regions. Thetenant-specific queues are configured for secure access by on-premisescloud regions through access credentials specific to a given on-premisescloud region. The cloud resource provisioning architecture also includesa public cloud queue for a public cloud region. The public cloud regionis configured to allow access by a provisioning workflow circuitry thatcommunicates provisioning instructions to multiple different publiccloud service providers.

The resource correlation circuitry is further configured to route thecommon resource provisioning template block to the public cloud queuefor retrieval by the public cloud provisioning workflow circuitry, whenthe common service provider region is the public cloud region. Theresource correlation circuitry is also configured to route the commonresource provisioning template block to the tenant-specific queue forretrieval by the on-premises cloud region, when the common serviceprovider region is the on-premises cloud region.

Expressed yet another way, the dispatcher circuitry 4108 calls the jobpreparation circuitry 4110 to initiate deployment of a resource set. Thedispatcher circuitry 4108 may, e.g., call a method defined by the jobpreparation circuitry 4110 and pass a URI to the template that containsdeployment instructions.

The job preparation circuitry 4110 uses the URI to retrieve thetemplate, e.g., from the template database 4114. The job preparationcircuitry 4110 breaks the template down (disaggregates them) into aseries of tasks. For instance, each resource may have its own task forindividual execution, e.g., by an automation script or runbook writtenfor the resource type.

The resource correlation circuitry 4112 re-aggregates resources based onpredefined correlation criteria. Multiple resources may be combined intotemplate blocks of correlated resources. The resource correlationcircuitry 4112 also decides whether the resources will be placed in apublic cloud region or an on-premises region, based on the regionresource characteristic associated with each resource. As noted above,there are tenant specific private cloud queues for on-premises regionsand public cloud queues for public regions. That is, the resourcecorrelation circuitry 4112 re-aggregates individual resources into alarger template (a template block), which are passed into the cloudqueue as a queued item. The provisioning workflow engines read themessages on their queues and determine that they specify provisioningaction. The provisioning workflow engines retrieve a script for thataction, and the input data for the script includes the template.

The script, e.g., a PowerShell script, performs the deployment. Forpublic cloud deployment using a template block, one advantage is thatthe public cloud provisioning workflow circuitry 4122 need not makecalls to native APIs to deploy the resources. Instead, the workflowcircuitry 4122 passes the template block through to the serviceprovider, and requests the service provider to use its native templatingability to cause instantiation of the resources in the template block.This avoids an implementation in which there are many different scriptsto write and maintain for each type of resource, and also avoidsexecuting multiple scripts when resources can be combined into atemplate block. Instead, the workflow circuitry 4122 passes the templateblock through to the native provider deployment process.

In one implementation, the resource correlation circuitry 4122 obtainsresource characteristics from the TTT circuitry 116. That is, the TTTcircuitry 116 may be extended to store and define (e.g., in the typedatabases 230) whether a type can be aggregated, and a script to executefor deploying a template block that includes the resource type. If theresource cannot be aggregated, then the resource correlation circuitry4122 keeps the resource separate, and sends a separate deploymentmessage for that uncorrelated resource to the cloud queue. One test forsetting the aggregate flag to true for a given type is that the serviceprovider can natively deploy the resource given a template for it.

The cloud resource provisioning architecture may work internally withtemplates having a given format. For instance, the architecture mayinternally use Azure Resource Manager (ARM) templates for specifyingresources to deploy individually or in a template block. Theprovisioning workflow engines 4122 and 4132 may deploy to regions hostedby many different service providers. In some implementations, theprovisioning workflow engines 4122 and 4132 may include templateconversion circuitry, e.g., the template conversion circuitry 4502 shownin FIG. 45.

The template conversion circuitry 4502 converts the internal templateformat, e.g., ARM templates, to the format used by the service providerwhere the resources will be deployed when the formats are incompatible.For instance, the template conversion circuitry 4502 may convert ARMtemplates to cloud formation templates (CFTs), or any other templateformat. The TTT circuitry 116 may provide the translation, e.g., byconverting resource types, such as an Azure VM, between serviceproviders, e.g., to an Amazon Web Services VM.

In the return direction, the provisioning workflow circuitry retrievesprovisioning messages saved on the cloud queues 4116 and 4118 by theresource correlation circuitry 4112. The provisioning messages specifywhat script the provisioning workflow circuitry should run, and what thedata payload is. The provisioning workflow circuitry retrieves thescript, passes the data payload to the script, and runs the script. Theprovisioning workflow circuitry need not know whether the data payloadis a template block, or a specifier of an individual resource.

On the other hand, the resource correlation circuitry 4122 knows whetherit has built a template block, and creates the provisioning message togive the instructions to the provisioning workflow engine to execute aspecific script for the resource type. As noted above, the TTT circuitry116 may be extended to include, as just one example set of resourcecharacteristics: 1) whether the resource type may be aggregated; 2) ifit can be aggregated then the URI of a script used to deploy a templateblock for a given service provider region; and 3) if the resource typecannot be aggregated, then the script for deploying the resource type asan uncorrelated resource.

For a template deployable resource, the script may simply pass thetemplate block to the service provider, and instruct the serviceprovider to instantiate the resources in the template block. In thatrespect, the template block may specify a single resource or multipleresources, each of which is template deployable by a given serviceprovider region. The resource correlation circuitry 4112 may specifynative service provider deployment for even a single resource, bypassing the template for the resource to the service provider, andrequesting the service provider to perform its native instantiationservice on the template. For non-template enabled resources, theprovisioning script may specify a sequence of calls to the native APIsof the service provider to provision the resource.

Note that even on-premises regions may deploy resources based ontemplate blocks. For instance, an on-premises version of Azure Stack™for software defined infrastructure may provide template deploymentfunctionality. Other template interpreters may be implemented to provideon-premises template deployment functionality.

The provisioning workflow circuitry 4122 and 4132 pass return messagesto the return queue 4120. The resource correlation circuitry 4112monitors the return queue 4120 to determine that the specifiedprovisioning actions have completed or failed. The return messages mayinclude a correlation identifier. The resource correlation circuitry4112 pulls return messages off of the return queue 4120, and sends themto the job preparation circuitry 4110 to inform the job preparationcircuitry 4110 that the deployment associates with the correlationidentifier is completed or failed. That is, the job preparationcircuitry 4110 tracks when each resource is deployed, and when allresources in a specific deployment are completed. The dispatchercircuitry 4108 polls the job preparation circuitry 4110 to determinewhen the whole deployment is complete or has failed, and provides thatstatus information back to the rest of the architecture.

The methods, devices, processing, circuitry, and logic described abovemay be implemented in many different ways and in many differentcombinations of hardware and software. For example, all or parts of theimplementations may be circuitry that includes an instruction processor,such as a Central Processing Unit (CPU), microcontroller, or amicroprocessor; or as an Application Specific Integrated Circuit (ASIC),Programmable Logic Device (PLD), or Field Programmable Gate Array(FPGA); or as circuitry that includes discrete logic or other circuitcomponents, including analog circuit components, digital circuitcomponents or both; or any combination thereof. The circuitry mayinclude discrete interconnected hardware components or may be combinedon a single integrated circuit die, distributed among multipleintegrated circuit dies, or implemented in a Multiple Chip Module (MCM)of multiple integrated circuit dies in a common package, as examples.

Accordingly, the circuitry may store or access instructions forexecution, or may implement its functionality in hardware alone. Theinstructions may be stored in a tangible storage medium that is otherthan a transitory signal, such as a flash memory, a Random Access Memory(RAM), a Read Only Memory (ROM), an Erasable Programmable Read OnlyMemory (EPROM); or on a magnetic or optical disc, such as a Compact DiscRead Only Memory (CDROM), Hard Disk Drive (HDD), or other magnetic oroptical disk; or in or on another machine-readable medium. A product,such as a computer program product, may include a storage medium andinstructions stored in or on the medium, and the instructions whenexecuted by the circuitry in a device may cause the device to implementany of the processing described above or illustrated in the drawings.

The implementations may be distributed. For instance, the circuitry mayinclude multiple distinct system components, such as multiple processorsand memories, and may span multiple distributed processing systems.Parameters, databases, and other data structures may be separatelystored and managed, may be incorporated into a single memory ordatabase, may be logically and physically organized in many differentways, and may be implemented in many different ways.

Example implementations include linked lists, program variables, hashtables, arrays, records (e.g., database records), objects, and implicitstorage mechanisms. Instructions may form parts (e.g., subroutines orother code sections) of a single program, may form multiple separateprograms, may be distributed across multiple memories and processors,and may be implemented in many different ways. Example implementationsinclude stand-alone programs, and as part of a library, such as a sharedlibrary like a Dynamic Link Library (DLL). The library, for example, maycontain shared data and one or more shared programs that includeinstructions that perform any of the processing described above orillustrated in the drawings, when executed by the circuitry.

Various implementations have been specifically described. However, manyother implementations are also possible.

What is claimed is:
 1. A system comprising: placement pipeline circuitrycomprising multiple processing stages, the placement pipeline circuitryconfigured to determine an initial placement location from initialplacement options for a technical component defined within a requesttemplate of a specified service request; timing metadata linked to thetechnical component and included within the request template, the timingmetadata defining a dynamic re-evaluation timing specifier forre-evaluating placement of the technical component; trigger metadatalinked to the technical component and included within the requesttemplate, the trigger metadata defining a dynamic re-evaluation triggercondition specifier for re-evaluating the placement of the technicalcomponent; and re-evaluation circuitry configured to, responsive to thedynamic re-evaluation timing specifier, the dynamic re-evaluationtrigger condition specifier, or both, re-execute the placement pipelinecircuitry on the technical component and determine updated placementoptions including a new placement location for the technical component;and wherein the placement pipeline circuitry comprises a subscriptionstage configured to: determine to a change to which service providerregions a resource requester has active subscriptions; and responsive tothe change, determine the new placement location.
 2. The system of claim1, where: the specified service request is linked to a specific resourcerequester; and further comprising: placement execution metadata for thespecific resource requester, the placement execution metadata definingan update time window for making adjustments to the specified servicerequest.
 3. The system of claim 2, where: the re-evaluation circuitry isfurther operable to initiate instantiation of a replacement componentfor the technical component at the new placement location responsive todetermining the updated placement options.
 4. The system of claim 3,where: the re-evaluation circuitry is further operable to initiateswitchover to the replacement component within the update time window.5. The system of claim 1, where: the placement pipeline circuitrycomprises a type stage configured to: determine a change to whichservice provider regions support the technical component; and responsiveto the change, determine the new placement location.
 6. The system ofclaim 5, where: the technical component comprises a technical componenttype specified in the request template for implementing the specifiedservice request.
 7. The system of claim 5 where: the technical componentcomprises a technical asset specified in the request template forimplementing the specified service request.
 8. The system of claim 7,where: the technical asset comprise a network.
 9. The system of claim 1,where: the placement pipeline circuitry comprises a capacity stageconfigured to: determine a change in which service provider regions havecapacity to provision the technical component; and responsive to thechange, determine the new placement location.
 10. The system of claim 1,where: the placement pipeline circuitry comprises a metadata stageconfigured to: determine a change to which service provider regionsqualify to provision the technical component; and responsive to thechange, determine the new placement location.
 11. A method comprising:in a hybrid cloud architect: receiving a technical service template forimplementing a service request for a resource requester; identifying aresource within the technical service template; executing, for theresource, placement pipeline circuitry comprising multiple processingstages configured to determine an initial placement from initialplacement options for the resource; obtaining, from within the hybridcloud architect, timing metadata linked to the resource and includedwithin the technical service template, the timing metadata defining atiming specifier for re-evaluating placement of the resource; obtaining,from within the hybrid cloud architect, trigger metadata linked to theresource and included within the technical service template, the triggermetadata defining a dynamic re-evaluation trigger condition specifierfor re-evaluating the placement of the resource; executing re-evaluationcircuitry configured to, responsive to the timing metadata, the triggermetadata, or both, determine when to re-execute the placement pipelinecircuitry for the resource and determine updated placement optionsincluding a new placement location for the resource; and at a metadataarchitecture of the hybrid cloud architect: determining a change towhich service provider regions qualify to provision the resource; andresponsive to the change, determining the new placement location. 12.The method of claim 11, further comprising: obtaining, from within thehybrid cloud architect, placement execution metadata linked to theresource requester, the placement execution metadata defining an updatetime window for implementing the new placement location.
 13. The methodof claim 12, further comprising: initiating provisioning of areplacement for the resource at the new placement location responsive todetermining the updated placement options.
 14. The method of claim 13,further comprising: initiating switchover to the replacement within theupdate time window.
 15. A system comprising: a communication interfaceconfigured to receive a technical service template configured toimplement a technical service request from a resource requester; ametadata architecture comprising: container metadata for a containerhierarchy defining a model of the resource requester; resource requestermetadata included within the technical service template, the resourcerequester metadata defining specific implementation data for thetechnical service request; and service provider metadata definingcapabilities of candidate service provider regions for hosting thetechnical service request; timing metadata linked to a computingresource and included within the technical service template, the timingmetadata defining a dynamic re-evaluation timing specifier forre-evaluating placement of the computing resource; trigger metadatalinked to the computing resource and included within the technicalservice template, the trigger metadata defining a dynamic re-evaluationtrigger condition specifier for re-evaluating placement of the computingresource; placement pipeline circuitry configured to: determine thecomputing resource defined within the technical service template, thecomputing resource for implementing the technical service request;receive the container metadata, the service provider metadata, and theresource requester metadata; execute sequential pipeline stages todetermine feasible placement options for the computing resource fromamong candidate placement options responsive to the container metadata,the service provider metadata, and the resource requester metadata;determine a change to which service provider regions qualify toprovision the resource; and responsive to the change, determine the newplacement location re-evaluation circuitry configured to, responsive tothe timing metadata, trigger metadata, or both, re-execute the placementpipeline circuitry on the computing resource and determine updatedplacement options including a new placement location for the computingresource.
 16. The system of claim 15, where: the re-evaluation circuitryis further configured to communicate the new placement location to theresource requester and request a re-placement confirmation from theresource requester.
 17. The system of claim 16, where: the re-evaluationcircuitry is further configured to, after receiving the re-placementconfirmation from the resource requester, schedule instantiation of asubstitute for the computing resource at the new placement location. 18.The system of claim 17, where: the re-evaluation circuitry is furtherconfigured to delay switchover to the substitute until an update timewindow is open for the resource requester.